2024-08-30T18:47:53.666 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-08-30T18:47:53.678 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-30T18:47:53.775 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-08-30_15:04:49-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7881988 branch: wip-yuri8-testing-2024-08-28-1632-squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7881988' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-08-30_15:04:49-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri8-testing-2024-08-28-1632-squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 workunit: branch: wip-yuri8-testing-2024-08-28-1632-squid sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2672 sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2024-08-28-1632-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYJ88z8Sthc0wPC9WGFYvCR0+pof4VgGhBBUzc99mk3aRs5UpWD6ISzU+Xpl83muJna4A4QkrlwApM1XRXimaY= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-30_15:04:49 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-30T18:47:53.775 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa; will attempt to use it 2024-08-30T18:47:53.776 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks 2024-08-30T18:47:53.776 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-30T18:47:53.778 INFO:teuthology.task.internal:Checking packages... 2024-08-30T18:47:53.800 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1ef864504b8875c83ee6c2c5fedc13315bebf7f5' 2024-08-30T18:47:53.800 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T18:47:53.800 INFO:teuthology.packaging:ref: None 2024-08-30T18:47:53.800 INFO:teuthology.packaging:tag: None 2024-08-30T18:47:53.800 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-08-28-1632-squid 2024-08-30T18:47:53.800 INFO:teuthology.packaging:sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-08-30T18:47:53.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri8-testing-2024-08-28-1632-squid 2024-08-30T18:47:53.949 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-235-g1ef86450-1jammy 2024-08-30T18:47:53.950 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-30T18:47:53.958 INFO:teuthology.task.internal:no buildpackages task found 2024-08-30T18:47:53.958 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-30T18:47:53.997 INFO:teuthology.task.internal:Saving configuration 2024-08-30T18:47:54.006 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-30T18:47:54.023 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-30T18:47:54.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-30_15:04:49-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7881988', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 18:42:19.464363', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDTQHWZjV+7MVZJJEUVIcN8WbD5RKmAhX6Wc1uAMqDDe'} 2024-08-30T18:47:54.053 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-30T18:47:54.060 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-30T18:47:54.060 INFO:teuthology.run_tasks:Running task console_log... 2024-08-30T18:47:54.123 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbbae093be0>, signals=[15]) 2024-08-30T18:47:54.123 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-30T18:47:54.129 INFO:teuthology.task.internal:Opening connections... 2024-08-30T18:47:54.129 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-08-30T18:47:54.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T18:47:54.220 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-30T18:47:54.226 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-08-30T18:47:54.317 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-08-30T18:47:54.318 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-08-30T18:47:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-08-30T18:47:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="Ubuntu" 2024-08-30T18:47:54.365 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="22.04" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_CODENAME=jammy 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:ID=ubuntu 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE=debian 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://www.ubuntu.com/" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-30T18:47:54.366 INFO:teuthology.orchestra.run.smithi079.stdout:UBUNTU_CODENAME=jammy 2024-08-30T18:47:54.367 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-08-30T18:47:54.396 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-30T18:47:54.404 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-30T18:47:54.440 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-30T18:47:54.440 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-08-30T18:47:54.445 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-30T18:47:54.464 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-30T18:47:54.465 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-08-30T18:47:54.496 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-30T18:47:54.537 INFO:teuthology.run_tasks:Running task kernel... 2024-08-30T18:47:54.551 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-30T18:47:54.551 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-30T18:47:54.551 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T18:47:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T18:47:54.556 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-08-30T18:47:54.602 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-56-generic 2024-08-30T18:47:54.602 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.15.0-56-generic 2024-08-30T18:47:54.602 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-08-30T18:47:54.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-08-30T18:47:54.859 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-30T18:47:54.880 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-08-30T18:47:54.943 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-08-30T18:47:54.979 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-08-30T18:47:55.086 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-08-30T18:47:55.159 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-08-30T18:47:55.194 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-08-30T18:47:55.197 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-08-30T18:47:55.197 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-08-30T18:47:55.197 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,384 kB] 2024-08-30T18:47:55.213 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-08-30T18:47:55.216 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2024-08-30T18:47:55.216 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [726 kB] 2024-08-30T18:47:55.221 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,112 kB] 2024-08-30T18:47:55.228 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [260 kB] 2024-08-30T18:47:55.229 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.0 kB] 2024-08-30T18:47:55.230 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-08-30T18:47:55.230 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-08-30T18:47:55.269 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-08-30T18:47:55.269 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-08-30T18:47:55.269 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-08-30T18:47:55.269 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-08-30T18:47:55.270 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-08-30T18:47:55.270 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-08-30T18:47:55.271 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-08-30T18:47:55.271 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-08-30T18:47:55.286 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-08-30T18:47:55.303 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-08-30T18:47:55.424 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,769 kB] 2024-08-30T18:47:55.840 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [531 kB] 2024-08-30T18:47:55.860 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [290 kB] 2024-08-30T18:47:55.873 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB] 2024-08-30T18:47:55.878 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-08-30T18:47:55.895 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,308 kB] 2024-08-30T18:47:55.962 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [398 kB] 2024-08-30T18:47:55.973 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [572 B] 2024-08-30T18:47:55.973 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB] 2024-08-30T18:47:56.005 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [622 kB] 2024-08-30T18:47:56.015 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB] 2024-08-30T18:47:56.022 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB] 2024-08-30T18:47:56.027 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-08-30T18:47:56.027 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-08-30T18:47:56.028 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-08-30T18:47:56.108 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-08-30T18:48:01.349 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 15.7 MB in 2s (7,538 kB/s) 2024-08-30T18:48:02.970 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:48:02.989 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-30T18:48:03.094 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:48:03.303 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-08-30T18:48:03.303 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-08-30T18:48:03.487 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-08-30T18:48:03.487 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-30T18:48:03.488 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-30T18:48:03.488 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-30T18:48:03.488 INFO:teuthology.orchestra.run.smithi079.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-08-30T18:48:03.488 INFO:teuthology.orchestra.run.smithi079.stdout: python2.7-minimal 2024-08-30T18:48:03.488 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-30T18:48:03.489 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-08-30T18:48:03.490 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-30T18:48:03.490 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-08-30T18:48:03.490 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-08-30T18:48:03.491 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-08-30T18:48:03.492 INFO:teuthology.orchestra.run.smithi079.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-08-30T18:48:03.492 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-08-30T18:48:03.492 INFO:teuthology.orchestra.run.smithi079.stdout: thermald 2024-08-30T18:48:03.544 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-08-30T18:48:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-08-30T18:48:03.544 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-08-30T18:48:03.545 INFO:teuthology.orchestra.run.smithi079.stdout: linux-modules-extra-5.15.0-119-generic 2024-08-30T18:48:03.546 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages will be upgraded: 2024-08-30T18:48:03.546 INFO:teuthology.orchestra.run.smithi079.stdout: linux-generic linux-headers-generic linux-image-generic 2024-08-30T18:48:03.727 INFO:teuthology.orchestra.run.smithi079.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-08-30T18:48:03.727 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 113 MB of archives. 2024-08-30T18:48:03.727 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 583 MB of additional disk space will be used. 2024-08-30T18:48:03.727 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-08-30T18:48:05.175 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-08-30T18:48:05.590 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-08-30T18:48:07.778 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-08-30T18:48:07.779 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-08-30T18:48:07.779 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-08-30T18:48:08.323 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-08-30T18:48:08.453 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-08-30T18:48:08.848 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 113 MB in 5s (23.1 MB/s) 2024-08-30T18:48:09.036 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-08-30T18:48:11.083 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-08-30T18:48:11.087 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-30T18:48:11.199 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:13.676 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-08-30T18:48:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-30T18:48:13.782 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:14.254 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-08-30T18:48:14.274 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-30T18:48:14.304 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:21.254 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-08-30T18:48:21.356 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-30T18:48:21.689 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-08-30T18:48:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-30T18:48:22.154 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-08-30T18:48:22.176 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-08-30T18:48:22.220 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-30T18:48:28.454 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-08-30T18:48:28.491 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-08-30T18:48:28.529 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:32.139 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-08-30T18:48:32.282 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-08-30T18:48:32.648 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-08-30T18:48:32.761 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:32.887 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-08-30T18:48:33.052 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:34.381 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-08-30T18:48:34.381 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-08-30T18:48:34.381 INFO:teuthology.orchestra.run.smithi079.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-08-30T18:48:34.381 INFO:teuthology.orchestra.run.smithi079.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-08-30T18:48:34.559 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:35.944 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:37.812 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-08-30T18:48:37.938 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-08-30T18:48:38.064 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-08-30T18:48:38.144 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-08-30T18:48:38.144 INFO:teuthology.orchestra.run.smithi079.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-08-30T18:48:52.131 INFO:teuthology.orchestra.run.smithi079.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-08-30T18:48:52.131 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub' 2024-08-30T18:48:52.154 INFO:teuthology.orchestra.run.smithi079.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-30T18:48:52.167 INFO:teuthology.orchestra.run.smithi079.stdout:Generating grub configuration file ... 2024-08-30T18:48:52.450 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-30T18:48:52.477 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-30T18:48:52.810 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-30T18:48:52.817 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-30T18:48:52.883 INFO:teuthology.orchestra.run.smithi079.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-30T18:48:52.888 INFO:teuthology.orchestra.run.smithi079.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-30T18:48:53.036 INFO:teuthology.orchestra.run.smithi079.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-30T18:48:53.036 INFO:teuthology.orchestra.run.smithi079.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-08-30T18:48:53.036 INFO:teuthology.orchestra.run.smithi079.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-30T18:48:53.079 INFO:teuthology.orchestra.run.smithi079.stdout:done 2024-08-30T18:48:53.560 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-08-30T18:48:53.580 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-08-30T18:48:53.580 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-08-30T18:48:53.580 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-08-30T18:48:53.580 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-08-30T18:48:53.580 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.119.119 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-08-30T18:48:53.581 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-08-30T18:48:53.581 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-08-30T18:48:53.582 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-30T18:48:53.582 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-30T18:48:53.582 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-30T18:48:53.582 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-08-30T18:48:53.582 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get clean 2024-08-30T18:48:53.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-08-30T18:48:53.858 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-30T18:48:53.867 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-30T18:48:53.884 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-30T18:48:53.978 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-30T18:48:55.270 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:48:55.288 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-30T18:48:55.344 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:48:55.539 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-08-30T18:48:55.540 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-08-30T18:48:55.705 INFO:teuthology.orchestra.run.smithi079.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-08-30T18:48:55.705 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-08-30T18:48:55.705 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-30T18:48:55.705 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-30T18:48:55.705 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-30T18:48:55.706 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-30T18:48:55.706 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-30T18:48:55.706 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-30T18:48:55.744 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-08-30T18:48:55.746 DEBUG:teuthology.orchestra.run.smithi079:> dpkg -s linux-image-generic 2024-08-30T18:48:55.761 INFO:teuthology.orchestra.run.smithi079.stdout:Package: linux-image-generic 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Status: install ok installed 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Priority: optional 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Section: kernel 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Installed-Size: 21 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Maintainer: Ubuntu Kernel Team 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: amd64 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Source: linux-meta 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Version: 5.15.0.119.119 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Recommends: thermald 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout:Description: Generic Linux kernel image 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout: This package will always depend on the latest generic kernel image 2024-08-30T18:48:55.762 INFO:teuthology.orchestra.run.smithi079.stdout: available. 2024-08-30T18:48:55.763 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-08-30T18:48:55.808 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.V0JOAm7AI1 2024-08-30T18:48:55.809 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub/grub.cfg /tmp/tmp.V0JOAm7AI1 2024-08-30T18:48:55.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.V0JOAm7AI1 2024-08-30T18:48:55.989 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.V0JOAm7AI1 is 10KB 2024-08-30T18:48:56.002 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.V0JOAm7AI1 2024-08-30T18:48:56.007 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-08-30T18:48:56.060 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:48:56.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-08-30T18:48:56.060 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-08-30T18:48:56.123 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-08-30T18:48:56.123 DEBUG:teuthology.orchestra.run.smithi079:> sudo update-grub 2024-08-30T18:48:57.464 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub' 2024-08-30T18:48:57.465 INFO:teuthology.orchestra.run.smithi079.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-30T18:48:57.467 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-08-30T18:48:57.694 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-08-30T18:48:57.699 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-08-30T18:48:57.940 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-30T18:48:57.946 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-30T18:48:57.988 INFO:teuthology.orchestra.run.smithi079.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-30T18:48:57.994 INFO:teuthology.orchestra.run.smithi079.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-30T18:48:58.162 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-30T18:48:58.162 INFO:teuthology.orchestra.run.smithi079.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-08-30T18:48:58.162 INFO:teuthology.orchestra.run.smithi079.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-30T18:48:58.183 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-08-30T18:48:58.185 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-08-30T18:49:28.217 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-30T18:49:28.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:49:28.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T18:50:02.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-08-30T18:50:11.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:50:11.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T18:50:30.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-08-30T18:50:42.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:50:42.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T18:50:43.096 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-08-30T18:50:43.835 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:50:43.835 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-08-30T18:50:43.835 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-08-30T18:50:43.880 INFO:teuthology.orchestra.run.smithi079.stdout:5.15.0-119-generic 2024-08-30T18:50:43.880 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-08-30T18:50:43.880 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-30T18:50:43.880 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-08-30T18:50:44.881 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-30T18:50:44.881 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T18:50:44.961 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-08-30T18:50:44.997 DEBUG:teuthology.parallel:result is None 2024-08-30T18:50:44.998 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-30T18:50:45.007 INFO:teuthology.task.internal:Creating test directory... 2024-08-30T18:50:45.007 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T18:50:45.013 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-30T18:50:45.045 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-30T18:50:45.095 INFO:teuthology.task.internal:Creating archive directory... 2024-08-30T18:50:45.095 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T18:50:45.113 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-30T18:50:45.137 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-30T18:50:45.137 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T18:50:45.182 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T18:50:45.190 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T18:50:45.192 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-30T18:50:45.199 INFO:teuthology.task.internal:Configuring sudo... 2024-08-30T18:50:45.199 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T18:50:45.246 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-30T18:50:45.256 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-30T18:50:45.256 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T18:50:45.292 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T18:50:45.340 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T18:50:45.388 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:50:45.388 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T18:50:45.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-08-30T18:50:45.527 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-30T18:50:45.534 INFO:teuthology.task.internal:Starting timer... 2024-08-30T18:50:45.534 INFO:teuthology.run_tasks:Running task pcp... 2024-08-30T18:50:45.573 INFO:teuthology.run_tasks:Running task selinux... 2024-08-30T18:50:45.615 DEBUG:teuthology.task.selinux:Excluding smithi079: OS 'ubuntu' does not support SELinux 2024-08-30T18:50:45.615 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-30T18:50:45.615 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-08-30T18:50:45.615 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-30T18:50:45.615 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-30T18:50:45.664 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-30T18:50:45.681 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-30T18:50:45.682 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-30T18:58:07.230 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com')] 2024-08-30T18:58:07.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:58:07.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T18:58:07.322 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-08-30T18:58:07.397 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-08-30T18:58:07.397 INFO:teuthology.run_tasks:Running task clock... 2024-08-30T18:58:07.409 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-30T18:58:07.410 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T18:58:07.410 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Command line: ntpd -gq 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: ---------------------------------------------------- 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: ntp-4 is maintained by Network Time Foundation, 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: corporation. Support and training for ntp-4 are 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: available at https://www.nwtime.org/support 2024-08-30T18:58:07.464 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: ---------------------------------------------------- 2024-08-30T18:58:07.465 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: proto: precision = 0.056 usec (-24) 2024-08-30T18:58:07.465 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: basedate set to 2022-02-04 2024-08-30T18:58:07.465 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: gps base set to 2022-02-06 (week 2196) 2024-08-30T18:58:07.465 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-30T18:58:07.466 INFO:teuthology.orchestra.run.smithi079.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-30T18:58:07.466 INFO:teuthology.orchestra.run.smithi079.stderr:restrict ::: KOD does nothing without LIMITED. 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: restrict ::: KOD does nothing without LIMITED. 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen and drop on 0 v6wildcard [::]:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen normally on 2 lo 127.0.0.1:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen normally on 3 enp3s0f1 172.21.15.79:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen normally on 4 lo [::1]:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce87%5]:123 2024-08-30T18:58:07.467 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:07 ntpd[18363]: Listening on routing socket on fd #22 for interface updates 2024-08-30T18:58:08.464 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:08.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:08 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:09.464 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:09.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:09.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:09.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:09.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:09.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:09 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:10.464 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:10.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:10.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:10.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:10.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:10.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:10 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:11.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:11.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:11 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:12.464 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:12.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:12.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:12.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:12.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:12.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:12 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:13.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:13.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:13.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:13 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: can't open /var/log/ntpstats/rawstats.20240830: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: can't open /var/log/ntpstats/peerstats.20240830: Permission denied 2024-08-30T18:58:14.465 INFO:teuthology.orchestra.run.smithi079.stdout:30 Aug 18:58:14 ntpd[18363]: ntpd: time slew -0.001759 s 2024-08-30T18:58:14.466 INFO:teuthology.orchestra.run.smithi079.stdout:ntpd: time slew -0.001759s 2024-08-30T18:58:14.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-08-30T18:58:14.466 INFO:teuthology.orchestra.run.smithi079.stderr:30 Aug 18:58:14 ntpd[18363]: can't open /var/log/ntpstats/loopstats.20240830: Permission denied 2024-08-30T18:58:14.517 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-08-30T18:58:14.517 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-08-30T18:58:14.518 INFO:teuthology.orchestra.run.smithi079.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-30T18:58:14.518 INFO:teuthology.orchestra.run.smithi079.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-30T18:58:14.518 INFO:teuthology.orchestra.run.smithi079.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-30T18:58:14.518 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-30T18:58:14.519 INFO:teuthology.run_tasks:Running task install... 2024-08-30T18:58:14.528 DEBUG:teuthology.task.install:project ceph 2024-08-30T18:58:14.528 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1ef864504b8875c83ee6c2c5fedc13315bebf7f5'}} 2024-08-30T18:58:14.528 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1ef864504b8875c83ee6c2c5fedc13315bebf7f5'} 2024-08-30T18:58:14.528 INFO:teuthology.task.install:Using flavor: default 2024-08-30T18:58:14.534 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-30T18:58:14.534 INFO:teuthology.task.install:extra packages: [] 2024-08-30T18:58:14.534 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-key list | grep Ceph 2024-08-30T18:58:14.617 INFO:teuthology.orchestra.run.smithi079.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-08-30T18:58:14.644 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-08-30T18:58:14.645 INFO:teuthology.orchestra.run.smithi079.stdout:uid [ unknown] Ceph.com (release key) 2024-08-30T18:58:14.645 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-08-30T18:58:14.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-08-30T18:58:14.806 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default/ 2024-08-30T18:58:14.806 INFO:teuthology.task.install.deb:Package version is 19.1.1-235-g1ef86450-1jammy 2024-08-30T18:58:14.928 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:58:14.928 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-08-30T18:58:14.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo apt-get update 2024-08-30T18:58:15.110 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-30T18:58:15.164 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy InRelease 2024-08-30T18:58:15.185 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-08-30T18:58:15.207 INFO:teuthology.orchestra.run.smithi079.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy Release.gpg 2024-08-30T18:58:15.228 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,405 B] 2024-08-30T18:58:15.236 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-30T18:58:15.257 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-08-30T18:58:15.311 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-30T18:58:15.393 INFO:teuthology.orchestra.run.smithi079.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-30T18:58:15.536 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 28.9 kB in 0s (70.4 kB/s) 2024-08-30T18:58:16.595 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:58:16.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-235-g1ef86450-1jammy cephadm=19.1.1-235-g1ef86450-1jammy ceph-mds=19.1.1-235-g1ef86450-1jammy ceph-mgr=19.1.1-235-g1ef86450-1jammy ceph-common=19.1.1-235-g1ef86450-1jammy ceph-fuse=19.1.1-235-g1ef86450-1jammy ceph-test=19.1.1-235-g1ef86450-1jammy ceph-volume=19.1.1-235-g1ef86450-1jammy radosgw=19.1.1-235-g1ef86450-1jammy python3-rados=19.1.1-235-g1ef86450-1jammy python3-rgw=19.1.1-235-g1ef86450-1jammy python3-cephfs=19.1.1-235-g1ef86450-1jammy python3-rbd=19.1.1-235-g1ef86450-1jammy libcephfs2=19.1.1-235-g1ef86450-1jammy libcephfs-dev=19.1.1-235-g1ef86450-1jammy librados2=19.1.1-235-g1ef86450-1jammy librbd1=19.1.1-235-g1ef86450-1jammy rbd-fuse=19.1.1-235-g1ef86450-1jammy 2024-08-30T18:58:16.666 INFO:teuthology.orchestra.run.smithi079.stdout:Reading package lists... 2024-08-30T18:58:16.851 INFO:teuthology.orchestra.run.smithi079.stdout:Building dependency tree... 2024-08-30T18:58:16.852 INFO:teuthology.orchestra.run.smithi079.stdout:Reading state information... 2024-08-30T18:58:17.038 INFO:teuthology.orchestra.run.smithi079.stdout:The following packages were automatically installed and are no longer required: 2024-08-30T18:58:17.038 INFO:teuthology.orchestra.run.smithi079.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-30T18:58:17.038 INFO:teuthology.orchestra.run.smithi079.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-30T18:58:17.038 INFO:teuthology.orchestra.run.smithi079.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-30T18:58:17.039 INFO:teuthology.orchestra.run.smithi079.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-30T18:58:17.039 INFO:teuthology.orchestra.run.smithi079.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-30T18:58:17.039 INFO:teuthology.orchestra.run.smithi079.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-30T18:58:17.040 INFO:teuthology.orchestra.run.smithi079.stdout:The following additional packages will be installed: 2024-08-30T18:58:17.040 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-08-30T18:58:17.040 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-08-30T18:58:17.040 INFO:teuthology.orchestra.run.smithi079.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-08-30T18:58:17.040 INFO:teuthology.orchestra.run.smithi079.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-08-30T18:58:17.041 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-08-30T18:58:17.042 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet zip 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout:Suggested packages: 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout:Recommended packages: 2024-08-30T18:58:17.043 INFO:teuthology.orchestra.run.smithi079.stdout: btrfs-tools 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout:The following NEW packages will be installed: 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-08-30T18:58:17.145 INFO:teuthology.orchestra.run.smithi079.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-08-30T18:58:17.146 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-08-30T18:58:17.147 INFO:teuthology.orchestra.run.smithi079.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-08-30T18:58:17.205 INFO:teuthology.orchestra.run.smithi079.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-08-30T18:58:17.205 INFO:teuthology.orchestra.run.smithi079.stdout:Need to get 181 MB of archives. 2024-08-30T18:58:17.205 INFO:teuthology.orchestra.run.smithi079.stdout:After this operation, 829 MB of additional disk space will be used. 2024-08-30T18:58:17.205 INFO:teuthology.orchestra.run.smithi079.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-08-30T18:58:17.288 INFO:teuthology.orchestra.run.smithi079.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-235-g1ef86450-1jammy [3,590 kB] 2024-08-30T18:58:17.291 INFO:teuthology.orchestra.run.smithi079.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-08-30T18:58:17.294 INFO:teuthology.orchestra.run.smithi079.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-08-30T18:58:17.311 INFO:teuthology.orchestra.run.smithi079.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-08-30T18:58:17.371 INFO:teuthology.orchestra.run.smithi079.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-08-30T18:58:17.375 INFO:teuthology.orchestra.run.smithi079.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-08-30T18:58:17.388 INFO:teuthology.orchestra.run.smithi079.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-08-30T18:58:17.392 INFO:teuthology.orchestra.run.smithi079.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-08-30T18:58:17.392 INFO:teuthology.orchestra.run.smithi079.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-08-30T18:58:17.393 INFO:teuthology.orchestra.run.smithi079.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-08-30T18:58:17.401 INFO:teuthology.orchestra.run.smithi079.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-08-30T18:58:17.404 INFO:teuthology.orchestra.run.smithi079.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-08-30T18:58:17.406 INFO:teuthology.orchestra.run.smithi079.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-08-30T18:58:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-08-30T18:58:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-08-30T18:58:17.412 INFO:teuthology.orchestra.run.smithi079.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-08-30T18:58:17.425 INFO:teuthology.orchestra.run.smithi079.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-08-30T18:58:17.427 INFO:teuthology.orchestra.run.smithi079.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-08-30T18:58:17.427 INFO:teuthology.orchestra.run.smithi079.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-08-30T18:58:17.428 INFO:teuthology.orchestra.run.smithi079.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-08-30T18:58:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-08-30T18:58:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-08-30T18:58:17.429 INFO:teuthology.orchestra.run.smithi079.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-08-30T18:58:17.430 INFO:teuthology.orchestra.run.smithi079.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-08-30T18:58:17.444 INFO:teuthology.orchestra.run.smithi079.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-08-30T18:58:17.444 INFO:teuthology.orchestra.run.smithi079.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-08-30T18:58:17.460 INFO:teuthology.orchestra.run.smithi079.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-08-30T18:58:17.463 INFO:teuthology.orchestra.run.smithi079.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-08-30T18:58:17.463 INFO:teuthology.orchestra.run.smithi079.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-08-30T18:58:17.464 INFO:teuthology.orchestra.run.smithi079.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-08-30T18:58:17.464 INFO:teuthology.orchestra.run.smithi079.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-08-30T18:58:17.466 INFO:teuthology.orchestra.run.smithi079.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-08-30T18:58:17.466 INFO:teuthology.orchestra.run.smithi079.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-08-30T18:58:17.466 INFO:teuthology.orchestra.run.smithi079.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-08-30T18:58:17.477 INFO:teuthology.orchestra.run.smithi079.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-08-30T18:58:17.478 INFO:teuthology.orchestra.run.smithi079.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-08-30T18:58:17.494 INFO:teuthology.orchestra.run.smithi079.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-08-30T18:58:17.500 INFO:teuthology.orchestra.run.smithi079.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-08-30T18:58:17.500 INFO:teuthology.orchestra.run.smithi079.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-08-30T18:58:17.500 INFO:teuthology.orchestra.run.smithi079.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-08-30T18:58:17.501 INFO:teuthology.orchestra.run.smithi079.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-08-30T18:58:17.502 INFO:teuthology.orchestra.run.smithi079.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-08-30T18:58:17.504 INFO:teuthology.orchestra.run.smithi079.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-08-30T18:58:17.505 INFO:teuthology.orchestra.run.smithi079.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-08-30T18:58:17.511 INFO:teuthology.orchestra.run.smithi079.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-08-30T18:58:17.512 INFO:teuthology.orchestra.run.smithi079.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-08-30T18:58:17.527 INFO:teuthology.orchestra.run.smithi079.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-235-g1ef86450-1jammy [3,240 kB] 2024-08-30T18:58:17.528 INFO:teuthology.orchestra.run.smithi079.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-08-30T18:58:17.529 INFO:teuthology.orchestra.run.smithi079.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-08-30T18:58:17.529 INFO:teuthology.orchestra.run.smithi079.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-08-30T18:58:17.532 INFO:teuthology.orchestra.run.smithi079.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-08-30T18:58:17.584 INFO:teuthology.orchestra.run.smithi079.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-08-30T18:58:17.585 INFO:teuthology.orchestra.run.smithi079.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-08-30T18:58:17.586 INFO:teuthology.orchestra.run.smithi079.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-08-30T18:58:17.587 INFO:teuthology.orchestra.run.smithi079.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-08-30T18:58:17.587 INFO:teuthology.orchestra.run.smithi079.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-08-30T18:58:17.605 INFO:teuthology.orchestra.run.smithi079.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-08-30T18:58:17.607 INFO:teuthology.orchestra.run.smithi079.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-08-30T18:58:17.607 INFO:teuthology.orchestra.run.smithi079.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-08-30T18:58:17.622 INFO:teuthology.orchestra.run.smithi079.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-08-30T18:58:17.622 INFO:teuthology.orchestra.run.smithi079.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-08-30T18:58:17.622 INFO:teuthology.orchestra.run.smithi079.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-08-30T18:58:17.623 INFO:teuthology.orchestra.run.smithi079.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-08-30T18:58:17.623 INFO:teuthology.orchestra.run.smithi079.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-08-30T18:58:17.624 INFO:teuthology.orchestra.run.smithi079.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-08-30T18:58:17.627 INFO:teuthology.orchestra.run.smithi079.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-235-g1ef86450-1jammy [719 kB] 2024-08-30T18:58:17.636 INFO:teuthology.orchestra.run.smithi079.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-08-30T18:58:17.637 INFO:teuthology.orchestra.run.smithi079.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-08-30T18:58:17.640 INFO:teuthology.orchestra.run.smithi079.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-08-30T18:58:17.648 INFO:teuthology.orchestra.run.smithi079.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-235-g1ef86450-1jammy [357 kB] 2024-08-30T18:58:17.653 INFO:teuthology.orchestra.run.smithi079.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-08-30T18:58:17.655 INFO:teuthology.orchestra.run.smithi079.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-08-30T18:58:17.655 INFO:teuthology.orchestra.run.smithi079.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-08-30T18:58:17.659 INFO:teuthology.orchestra.run.smithi079.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-08-30T18:58:17.661 INFO:teuthology.orchestra.run.smithi079.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-08-30T18:58:17.662 INFO:teuthology.orchestra.run.smithi079.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-235-g1ef86450-1jammy [32.8 kB] 2024-08-30T18:58:17.662 INFO:teuthology.orchestra.run.smithi079.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-08-30T18:58:17.662 INFO:teuthology.orchestra.run.smithi079.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-08-30T18:58:17.663 INFO:teuthology.orchestra.run.smithi079.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-235-g1ef86450-1jammy [180 kB] 2024-08-30T18:58:17.666 INFO:teuthology.orchestra.run.smithi079.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-235-g1ef86450-1jammy [67.6 kB] 2024-08-30T18:58:17.668 INFO:teuthology.orchestra.run.smithi079.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-235-g1ef86450-1jammy [333 kB] 2024-08-30T18:58:17.670 INFO:teuthology.orchestra.run.smithi079.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-08-30T18:58:17.671 INFO:teuthology.orchestra.run.smithi079.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-08-30T18:58:17.673 INFO:teuthology.orchestra.run.smithi079.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-08-30T18:58:17.679 INFO:teuthology.orchestra.run.smithi079.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-235-g1ef86450-1jammy [6,874 kB] 2024-08-30T18:58:17.687 INFO:teuthology.orchestra.run.smithi079.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-08-30T18:58:17.688 INFO:teuthology.orchestra.run.smithi079.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-08-30T18:58:17.688 INFO:teuthology.orchestra.run.smithi079.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-08-30T18:58:17.689 INFO:teuthology.orchestra.run.smithi079.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-08-30T18:58:17.690 INFO:teuthology.orchestra.run.smithi079.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-08-30T18:58:17.696 INFO:teuthology.orchestra.run.smithi079.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-08-30T18:58:17.696 INFO:teuthology.orchestra.run.smithi079.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-08-30T18:58:17.704 INFO:teuthology.orchestra.run.smithi079.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-08-30T18:58:17.706 INFO:teuthology.orchestra.run.smithi079.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-08-30T18:58:17.706 INFO:teuthology.orchestra.run.smithi079.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-08-30T18:58:17.829 INFO:teuthology.orchestra.run.smithi079.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-235-g1ef86450-1jammy [112 kB] 2024-08-30T18:58:17.831 INFO:teuthology.orchestra.run.smithi079.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-235-g1ef86450-1jammy [472 kB] 2024-08-30T18:58:17.842 INFO:teuthology.orchestra.run.smithi079.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-235-g1ef86450-1jammy [25.9 MB] 2024-08-30T18:58:18.880 INFO:teuthology.orchestra.run.smithi079.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-235-g1ef86450-1jammy [5,144 kB] 2024-08-30T18:58:18.955 INFO:teuthology.orchestra.run.smithi079.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-235-g1ef86450-1jammy [245 kB] 2024-08-30T18:58:18.958 INFO:teuthology.orchestra.run.smithi079.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-235-g1ef86450-1jammy [124 kB] 2024-08-30T18:58:18.961 INFO:teuthology.orchestra.run.smithi079.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-235-g1ef86450-1jammy [1,555 kB] 2024-08-30T18:58:18.990 INFO:teuthology.orchestra.run.smithi079.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-235-g1ef86450-1jammy [6,127 kB] 2024-08-30T18:58:19.108 INFO:teuthology.orchestra.run.smithi079.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-235-g1ef86450-1jammy [22.8 MB] 2024-08-30T18:58:19.653 INFO:teuthology.orchestra.run.smithi079.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-235-g1ef86450-1jammy [14.1 kB] 2024-08-30T18:58:19.703 INFO:teuthology.orchestra.run.smithi079.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-235-g1ef86450-1jammy [877 kB] 2024-08-30T18:58:19.716 INFO:teuthology.orchestra.run.smithi079.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-235-g1ef86450-1jammy [2,462 kB] 2024-08-30T18:58:19.750 INFO:teuthology.orchestra.run.smithi079.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-235-g1ef86450-1jammy [794 kB] 2024-08-30T18:58:19.760 INFO:teuthology.orchestra.run.smithi079.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-235-g1ef86450-1jammy [152 kB] 2024-08-30T18:58:19.762 INFO:teuthology.orchestra.run.smithi079.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-235-g1ef86450-1jammy [2,323 kB] 2024-08-30T18:58:19.784 INFO:teuthology.orchestra.run.smithi079.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-235-g1ef86450-1jammy [8,625 kB] 2024-08-30T18:58:22.513 INFO:teuthology.orchestra.run.smithi079.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-235-g1ef86450-1jammy [14.2 kB] 2024-08-30T18:58:22.532 INFO:teuthology.orchestra.run.smithi079.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-235-g1ef86450-1jammy [51.3 MB] 2024-08-30T18:58:24.615 INFO:teuthology.orchestra.run.smithi079.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-235-g1ef86450-1jammy [130 kB] 2024-08-30T18:58:24.617 INFO:teuthology.orchestra.run.smithi079.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-235-g1ef86450-1jammy [39.0 kB] 2024-08-30T18:58:24.619 INFO:teuthology.orchestra.run.smithi079.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-235-g1ef86450-1jammy [13.6 MB] 2024-08-30T18:58:24.906 INFO:teuthology.orchestra.run.smithi079.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-235-g1ef86450-1jammy [91.6 kB] 2024-08-30T18:58:25.474 INFO:teuthology.orchestra.run.smithi079.stdout:Fetched 181 MB in 8s (23.3 MB/s) 2024-08-30T18:58:25.639 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-08-30T18:58:25.737 INFO:teuthology.orchestra.run.smithi079.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-08-30T18:58:25.742 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-08-30T18:58:25.821 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-30T18:58:26.166 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-08-30T18:58:26.193 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-08-30T18:58:26.233 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-30T18:58:26.562 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-08-30T18:58:26.587 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-08-30T18:58:26.628 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-30T18:58:27.123 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-08-30T18:58:27.149 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-30T18:58:27.197 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:58:27.609 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-08-30T18:58:27.635 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-30T18:58:27.676 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:58:28.012 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-08-30T18:58:28.039 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-30T18:58:28.078 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:58:28.407 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-08-30T18:58:28.436 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-08-30T18:58:28.473 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-30T18:58:28.785 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librados2. 2024-08-30T18:58:28.810 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../007-librados2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:28.851 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librados2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:29.312 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librbd1. 2024-08-30T18:58:29.338 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../008-librbd1_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:29.379 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librbd1 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:29.790 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs2. 2024-08-30T18:58:29.816 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:29.848 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:30.109 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rados. 2024-08-30T18:58:30.125 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../010-python3-rados_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:30.167 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rados (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:30.486 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-08-30T18:58:30.512 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:58:30.553 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-argparse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:30.830 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cephfs. 2024-08-30T18:58:30.845 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:30.897 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cephfs (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:31.167 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-ceph-common. 2024-08-30T18:58:31.182 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:58:31.216 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:31.511 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-wcwidth. 2024-08-30T18:58:31.537 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-08-30T18:58:31.578 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-30T18:58:31.889 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-prettytable. 2024-08-30T18:58:31.916 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-08-30T18:58:31.955 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-08-30T18:58:32.208 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rbd. 2024-08-30T18:58:32.223 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:32.266 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rbd (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:32.628 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-08-30T18:58:32.654 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-08-30T18:58:32.694 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-30T18:58:32.971 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-08-30T18:58:32.986 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-08-30T18:58:33.021 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-08-30T18:58:33.300 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-08-30T18:58:33.326 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-08-30T18:58:33.366 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-30T18:58:33.694 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua5.1. 2024-08-30T18:58:33.720 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-08-30T18:58:33.761 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-08-30T18:58:34.080 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-any. 2024-08-30T18:58:34.106 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-08-30T18:58:34.156 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-any (27ubuntu1) ... 2024-08-30T18:58:34.374 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package zip. 2024-08-30T18:58:34.389 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-08-30T18:58:34.425 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking zip (3.0-12build2) ... 2024-08-30T18:58:34.753 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package unzip. 2024-08-30T18:58:34.779 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-08-30T18:58:34.811 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-08-30T18:58:35.190 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package luarocks. 2024-08-30T18:58:35.216 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-08-30T18:58:35.256 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-08-30T18:58:35.639 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package librgw2. 2024-08-30T18:58:35.665 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../025-librgw2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:35.714 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking librgw2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:36.541 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rgw. 2024-08-30T18:58:36.557 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:36.591 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rgw (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:36.869 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package liboath0:amd64. 2024-08-30T18:58:36.895 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-08-30T18:58:36.936 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-08-30T18:58:37.239 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libradosstriper1. 2024-08-30T18:58:37.265 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:37.305 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libradosstriper1 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:37.734 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-common. 2024-08-30T18:58:37.760 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../029-ceph-common_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:37.800 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:39.164 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-base. 2024-08-30T18:58:39.190 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../030-ceph-base_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:39.239 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-base (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:39.793 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-08-30T18:58:39.819 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-08-30T18:58:39.851 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-08-30T18:58:40.137 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cheroot. 2024-08-30T18:58:40.164 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-08-30T18:58:40.204 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-30T18:58:40.532 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-08-30T18:58:40.558 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-08-30T18:58:40.599 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-08-30T18:58:40.901 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.text. 2024-08-30T18:58:40.927 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-08-30T18:58:40.968 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-08-30T18:58:41.229 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-08-30T18:58:41.244 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-08-30T18:58:41.279 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-08-30T18:58:41.559 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempora. 2024-08-30T18:58:41.585 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-08-30T18:58:41.625 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-08-30T18:58:41.929 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-portend. 2024-08-30T18:58:41.955 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-08-30T18:58:41.995 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-08-30T18:58:42.257 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-08-30T18:58:42.283 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-08-30T18:58:42.323 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-08-30T18:58:42.635 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cherrypy3. 2024-08-30T18:58:42.661 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-08-30T18:58:42.701 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-08-30T18:58:43.047 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-natsort. 2024-08-30T18:58:43.062 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-08-30T18:58:43.097 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-08-30T18:58:43.417 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-logutils. 2024-08-30T18:58:43.443 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-08-30T18:58:43.484 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-08-30T18:58:43.787 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-markupsafe. 2024-08-30T18:58:43.813 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-08-30T18:58:43.853 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-08-30T18:58:44.156 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-mako. 2024-08-30T18:58:44.183 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-08-30T18:58:44.223 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-30T18:58:44.527 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplegeneric. 2024-08-30T18:58:44.553 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-08-30T18:58:44.585 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-08-30T18:58:44.872 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-singledispatch. 2024-08-30T18:58:44.898 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-08-30T18:58:44.938 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-08-30T18:58:45.249 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webob. 2024-08-30T18:58:45.276 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-08-30T18:58:45.316 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-08-30T18:58:45.653 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-waitress. 2024-08-30T18:58:45.679 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-08-30T18:58:45.722 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-30T18:58:46.031 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-tempita. 2024-08-30T18:58:46.057 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-08-30T18:58:46.097 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-08-30T18:58:46.409 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-paste. 2024-08-30T18:58:46.436 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-08-30T18:58:46.475 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-08-30T18:58:46.787 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-08-30T18:58:46.803 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-08-30T18:58:46.845 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-08-30T18:58:47.157 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastedeploy. 2024-08-30T18:58:47.183 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-08-30T18:58:47.223 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-08-30T18:58:47.502 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-webtest. 2024-08-30T18:58:47.529 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-08-30T18:58:47.568 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-08-30T18:58:47.838 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pecan. 2024-08-30T18:58:47.853 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-08-30T18:58:47.896 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-08-30T18:58:48.227 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-certifi. 2024-08-30T18:58:48.254 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-08-30T18:58:48.293 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-08-30T18:58:48.613 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-idna. 2024-08-30T18:58:48.640 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-08-30T18:58:48.680 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-08-30T18:58:48.991 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-urllib3. 2024-08-30T18:58:49.018 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-08-30T18:58:49.066 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-30T18:58:49.386 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests. 2024-08-30T18:58:49.413 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-08-30T18:58:49.453 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-30T18:58:49.731 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-werkzeug. 2024-08-30T18:58:49.758 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-08-30T18:58:49.789 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-30T18:58:50.068 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-08-30T18:58:50.095 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:58:50.134 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-modules-core (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:51.466 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-08-30T18:58:51.492 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:51.532 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:52.052 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-bcrypt. 2024-08-30T18:58:52.078 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-08-30T18:58:52.118 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-08-30T18:58:52.397 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-openssl. 2024-08-30T18:58:52.423 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-08-30T18:58:52.463 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-08-30T18:58:52.800 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr. 2024-08-30T18:58:52.826 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:52.867 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:53.228 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mon. 2024-08-30T18:58:53.255 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:53.294 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mon (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:53.873 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-osd. 2024-08-30T18:58:53.900 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:53.940 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-osd (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:54.993 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph. 2024-08-30T18:58:55.009 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../066-ceph_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:55.051 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:55.380 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-fuse. 2024-08-30T18:58:55.406 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:55.438 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:55.800 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mds. 2024-08-30T18:58:55.827 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:55.866 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mds (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:56.269 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package cephadm. 2024-08-30T18:58:56.296 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../069-cephadm_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:58:56.336 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:56.639 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-asyncssh. 2024-08-30T18:58:56.666 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-08-30T18:58:56.722 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-08-30T18:58:57.042 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-babel-localedata. 2024-08-30T18:58:57.069 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-08-30T18:58:57.109 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-30T18:58:58.072 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-babel. 2024-08-30T18:58:58.100 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-08-30T18:58:58.139 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-08-30T18:58:58.417 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-jinja2. 2024-08-30T18:58:58.444 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-08-30T18:58:58.475 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-30T18:58:58.804 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-08-30T18:58:58.831 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:58:58.870 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:58:59.215 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-repoze.lru. 2024-08-30T18:58:59.242 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-08-30T18:58:59.315 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-08-30T18:58:59.619 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-routes. 2024-08-30T18:58:59.645 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-08-30T18:58:59.685 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-08-30T18:58:59.988 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-08-30T18:59:00.015 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:59:00.055 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-dashboard (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:01.352 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-08-30T18:59:01.379 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-08-30T18:59:01.419 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-30T18:59:01.914 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-joblib. 2024-08-30T18:59:01.941 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-08-30T18:59:01.980 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-08-30T18:59:02.351 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-08-30T18:59:02.378 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-08-30T18:59:02.418 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-08-30T18:59:02.695 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-sklearn. 2024-08-30T18:59:02.711 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-08-30T18:59:02.745 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-30T18:59:03.430 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-08-30T18:59:03.457 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:59:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:04.536 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-cachetools. 2024-08-30T18:59:04.563 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-08-30T18:59:04.603 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-08-30T18:59:04.864 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1. 2024-08-30T18:59:04.879 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-08-30T18:59:04.914 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-08-30T18:59:05.218 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-08-30T18:59:05.245 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-08-30T18:59:05.284 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-08-30T18:59:05.588 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-rsa. 2024-08-30T18:59:05.615 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-08-30T18:59:05.654 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-rsa (4.8-1) ... 2024-08-30T18:59:05.975 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-google-auth. 2024-08-30T18:59:06.002 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-08-30T18:59:06.041 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-08-30T18:59:06.362 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-08-30T18:59:06.389 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-08-30T18:59:06.428 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-30T18:59:06.748 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-websocket. 2024-08-30T18:59:06.775 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-08-30T18:59:06.814 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-08-30T18:59:07.302 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-kubernetes. 2024-08-30T18:59:07.329 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-08-30T18:59:07.368 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-30T18:59:08.016 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-08-30T18:59:08.032 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:59:08.074 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:08.402 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libonig5:amd64. 2024-08-30T18:59:08.429 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-08-30T18:59:08.469 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-30T18:59:08.806 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libjq1:amd64. 2024-08-30T18:59:08.833 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-08-30T18:59:08.872 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-30T18:59:09.108 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package jq. 2024-08-30T18:59:09.124 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-08-30T18:59:09.166 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-08-30T18:59:09.420 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package socat. 2024-08-30T18:59:09.447 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-08-30T18:59:09.486 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-08-30T18:59:09.806 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package xmlstarlet. 2024-08-30T18:59:09.822 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-08-30T18:59:09.864 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-08-30T18:59:10.211 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-test. 2024-08-30T18:59:10.227 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../097-ceph-test_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:59:10.269 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-test (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:12.632 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package ceph-volume. 2024-08-30T18:59:12.659 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-08-30T18:59:12.698 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking ceph-volume (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package libcephfs-dev. 2024-08-30T18:59:13.027 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:59:13.069 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking libcephfs-dev (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:13.372 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-socket:amd64. 2024-08-30T18:59:13.399 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-08-30T18:59:13.439 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-30T18:59:13.761 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package lua-sec:amd64. 2024-08-30T18:59:13.788 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-08-30T18:59:13.827 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-08-30T18:59:14.123 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package nvme-cli. 2024-08-30T18:59:14.150 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-08-30T18:59:14.181 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-08-30T18:59:14.579 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package pkg-config. 2024-08-30T18:59:14.606 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-08-30T18:59:14.645 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-08-30T18:59:14.890 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-08-30T18:59:14.917 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-08-30T18:59:14.956 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-08-30T18:59:15.343 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-iniconfig. 2024-08-30T18:59:15.359 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-08-30T18:59:15.393 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-08-30T18:59:15.721 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pastescript. 2024-08-30T18:59:15.749 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-08-30T18:59:15.787 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-08-30T18:59:16.108 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pluggy. 2024-08-30T18:59:16.124 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-08-30T18:59:16.175 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-08-30T18:59:16.486 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-psutil. 2024-08-30T18:59:16.513 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-08-30T18:59:16.552 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-08-30T18:59:16.881 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-py. 2024-08-30T18:59:16.908 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-08-30T18:59:16.955 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-py (1.10.0-1) ... 2024-08-30T18:59:17.293 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pygments. 2024-08-30T18:59:17.320 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-08-30T18:59:17.359 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-08-30T18:59:17.831 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pyinotify. 2024-08-30T18:59:17.858 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-08-30T18:59:17.897 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-08-30T18:59:18.200 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-toml. 2024-08-30T18:59:18.227 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-08-30T18:59:18.268 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-08-30T18:59:18.565 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-pytest. 2024-08-30T18:59:18.568 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-08-30T18:59:18.600 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-08-30T18:59:18.898 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package python3-simplejson. 2024-08-30T18:59:18.925 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-08-30T18:59:18.965 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-08-30T18:59:19.251 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-08-30T18:59:19.278 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-08-30T18:59:19.317 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-08-30T18:59:20.047 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package radosgw. 2024-08-30T18:59:20.075 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../116-radosgw_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:59:20.113 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking radosgw (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:20.944 INFO:teuthology.orchestra.run.smithi079.stdout:Selecting previously unselected package rbd-fuse. 2024-08-30T18:59:20.959 INFO:teuthology.orchestra.run.smithi079.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-08-30T18:59:21.001 INFO:teuthology.orchestra.run.smithi079.stdout:Unpacking rbd-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:21.265 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-08-30T18:59:21.516 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-30T18:59:21.642 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-08-30T18:59:21.906 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-08-30T18:59:22.220 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-08-30T18:59:22.612 INFO:teuthology.orchestra.run.smithi079.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-08-30T18:59:22.685 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:23.160 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user cephadm....done 2024-08-30T18:59:23.309 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-30T18:59:23.594 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-08-30T18:59:23.870 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-08-30T18:59:23.971 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-08-30T18:59:24.208 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-08-30T18:59:24.465 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-08-30T18:59:24.592 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-py (1.10.0-1) ... 2024-08-30T18:59:24.894 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-08-30T18:59:25.254 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-08-30T18:59:25.513 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-08-30T18:59:25.773 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-08-30T18:59:26.016 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-08-30T18:59:26.276 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-08-30T18:59:26.535 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-08-30T18:59:26.795 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-argparse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:27.054 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-30T18:59:27.231 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-30T18:59:27.457 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-08-30T18:59:27.583 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-08-30T18:59:27.842 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-08-30T18:59:28.231 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-08-30T18:59:28.282 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-08-30T18:59:28.346 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-30T18:59:28.447 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-08-30T18:59:28.798 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-30T18:59:28.925 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-08-30T18:59:29.209 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-08-30T18:59:29.486 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-08-30T18:59:29.787 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up zip (3.0-12build2) ... 2024-08-30T18:59:29.896 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-08-30T18:59:30.565 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-08-30T18:59:30.816 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-08-30T18:59:30.942 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-08-30T18:59:31.077 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-30T18:59:31.403 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-08-30T18:59:31.787 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-08-30T18:59:32.013 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-08-30T18:59:32.416 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-30T18:59:32.717 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-30T18:59:33.077 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-08-30T18:59:33.328 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-08-30T18:59:33.454 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:33.764 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-08-30T18:59:34.090 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-30T18:59:35.246 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-08-30T18:59:35.498 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:59:35.632 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-toml (0.10.2-1) ... 2024-08-30T18:59:35.892 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-30T18:59:36.018 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-08-30T18:59:36.169 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-30T18:59:36.463 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-08-30T18:59:36.705 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-08-30T18:59:36.931 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:59:37.057 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-08-30T18:59:37.359 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-08-30T18:59:37.610 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-08-30T18:59:37.887 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-08-30T18:59:38.404 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-08-30T18:59:38.656 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-08-30T18:59:38.915 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-30T18:59:39.049 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-08-30T18:59:39.330 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-30T18:59:39.452 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-30T18:59:39.737 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-08-30T18:59:40.013 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-08-30T18:59:40.273 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-30T18:59:40.399 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-08-30T18:59:40.525 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-08-30T18:59:40.622 INFO:teuthology.orchestra.run.smithi079.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-08-30T18:59:40.860 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-30T18:59:40.986 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-08-30T18:59:41.363 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-08-30T18:59:41.605 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up lua-any (27ubuntu1) ... 2024-08-30T18:59:41.732 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-portend (3.0.0-1) ... 2024-08-30T18:59:41.983 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-30T18:59:42.109 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-30T18:59:42.427 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-08-30T18:59:42.736 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-30T18:59:42.988 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-08-30T18:59:43.114 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rsa (4.8-1) ... 2024-08-30T18:59:43.381 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-08-30T18:59:43.649 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-30T18:59:44.003 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-08-30T18:59:44.372 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-08-30T18:59:44.657 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-08-30T18:59:44.993 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-30T18:59:45.119 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librados2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:45.245 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libsqlite3-mod-ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:45.363 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-08-30T18:59:45.699 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:45.826 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libradosstriper1 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:45.952 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-08-30T18:59:46.220 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librbd1 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:46.321 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-modules-core (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:46.448 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:46.659 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-30T18:59:46.659 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-30T18:59:47.116 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up libcephfs-dev (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:47.250 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rados (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:47.376 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up librgw2 (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:47.502 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-30T18:59:48.698 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rbd (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:48.824 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up rbd-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:48.959 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-rgw (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:49.085 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up python3-cephfs (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:49.211 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:49.729 INFO:teuthology.orchestra.run.smithi079.stdout:Adding group ceph....done 2024-08-30T18:59:50.114 INFO:teuthology.orchestra.run.smithi079.stdout:Adding system user ceph....done 2024-08-30T18:59:50.282 INFO:teuthology.orchestra.run.smithi079.stdout:Setting system user ceph properties....done 2024-08-30T18:59:50.302 INFO:teuthology.orchestra.run.smithi079.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-08-30T18:59:50.443 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-08-30T18:59:50.794 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-08-30T18:59:51.238 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-test (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:51.364 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up radosgw (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:51.843 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-30T18:59:51.843 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-30T18:59:52.308 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-base (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:52.664 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-08-30T18:59:53.143 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mds (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:53.366 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-30T18:59:53.366 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-30T18:59:53.828 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:54.047 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-30T18:59:54.048 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-30T18:59:54.513 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-osd (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:54.784 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-30T18:59:54.784 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-30T18:59:55.281 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-k8sevents (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:55.407 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:55.567 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mon (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:55.780 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-30T18:59:55.780 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-30T18:59:56.310 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:56.445 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:57.063 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-mgr-dashboard (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:57.240 INFO:teuthology.orchestra.run.smithi079.stdout:Setting up ceph-volume (19.1.1-235-g1ef86450-1jammy) ... 2024-08-30T18:59:57.593 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-08-30T18:59:57.702 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-08-30T18:59:57.846 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-08-30T18:59:58.982 INFO:teuthology.orchestra.run.smithi079.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-08-30T18:59:59.626 INFO:teuthology.orchestra.run.smithi079.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-08-30T18:59:59.631 DEBUG:teuthology.parallel:result is None 2024-08-30T18:59:59.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-08-30T18:59:59.792 DEBUG:teuthology.orchestra.run.smithi079:> dpkg-query -W -f '${Version}' ceph 2024-08-30T18:59:59.810 INFO:teuthology.orchestra.run.smithi079.stdout:19.1.1-235-g1ef86450-1jammy 2024-08-30T18:59:59.811 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-235-g1ef86450-1jammy 2024-08-30T18:59:59.811 INFO:teuthology.task.install:The correct ceph version 19.1.1-235-g1ef86450-1jammy is installed. 2024-08-30T18:59:59.813 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-30T18:59:59.813 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:59:59.813 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T18:59:59.865 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-30T18:59:59.865 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:59:59.865 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T18:59:59.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T18:59:59.975 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-30T18:59:59.976 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T18:59:59.976 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T19:00:00.036 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T19:00:00.091 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-30T19:00:00.091 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T19:00:00.091 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T19:00:00.149 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T19:00:00.202 INFO:teuthology.run_tasks:Running task workunit... 2024-08-30T19:00:00.219 INFO:tasks.workunit:Pulling workunits from ref 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-08-30T19:00:00.219 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-30T19:00:00.220 INFO:tasks.workunit:timeout=3h 2024-08-30T19:00:00.220 INFO:tasks.workunit:cleanup=True 2024-08-30T19:00:00.220 DEBUG:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-30T19:00:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T19:00:00.248 INFO:teuthology.orchestra.run.smithi079.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-30T19:00:00.248 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-30T19:00:00.296 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-30T19:00:00.296 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-30T19:00:00.344 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-08-30T19:00:00.394 INFO:tasks.workunit.client.0.smithi079.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-30T19:00:46.281 INFO:tasks.workunit.client.0.smithi079.stderr:Note: switching to '1ef864504b8875c83ee6c2c5fedc13315bebf7f5'. 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by switching back to a branch. 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: git switch -c 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:Or undo this operation with: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.282 INFO:tasks.workunit.client.0.smithi079.stderr: git switch - 2024-08-30T19:00:46.283 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.283 INFO:tasks.workunit.client.0.smithi079.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-30T19:00:46.283 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-08-30T19:00:46.283 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 1ef864504b8 Merge branch 'wip-bz-2305110-squid' of https://github.com/ljflores/ceph into wip-yuri8-testing-2024-08-28-1632-squid 2024-08-30T19:00:46.288 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-30T19:00:46.337 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-08-30T19:00:46.337 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-30T19:00:46.383 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-08-30T19:00:46.384 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-08-30T19:00:46.384 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-08-30T19:00:46.449 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-30T19:00:46.480 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-30T19:00:46.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:00:46.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/bad-inc-map 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:39527 2024-08-30T19:00:46.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:39527 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none ' 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:39527 ' 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-08-30T19:00:46.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-08-30T19:00:46.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:00:46.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:00:46.486 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:00:46.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:00:46.486 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:00:46.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:00:46.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:00:46.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:00:46.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:00:46.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:00:46.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:00:46.489 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:00:46.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:00:46.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:00:46.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-30T19:00:46.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:00:46.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:46.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:46.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21391 2024-08-30T19:00:46.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:00:46.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:00:46.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-08-30T19:00:46.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:00:46.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:46.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:46.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21391 2024-08-30T19:00:46.496 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:00:46.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-08-30T19:00:46.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-08-30T19:00:46.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:00:46.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:00:46.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:00:46.966 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:00:46.966 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:46.966 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:46.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:00:46.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:00:47.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:00:47.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:00:47.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:00:47.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:00:47.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:00:47.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:00:47.166 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:00:47.166 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:00:47.166 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:00:47.166 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:00:47.166 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:47.167 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:47.167 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21391/ceph-mon.a.asok 2024-08-30T19:00:47.167 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:00:47.167 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21391/ceph-mon.a.asok config get fsid 2024-08-30T19:00:47.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:00:47.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:00:47.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:00:47.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:00:47.286 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:47.287 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:47.287 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21391/ceph-mon.a.asok 2024-08-30T19:00:47.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:00:47.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21391/ceph-mon.a.asok config get mon_host 2024-08-30T19:00:47.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-08-30T19:00:47.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-08-30T19:00:47.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:00:47.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:00:47.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:00:47.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-08-30T19:00:47.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:00:48.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:00:48.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:00:48.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:00:48.303 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:00:48.303 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:48.303 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:48.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:00:48.304 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:00:48.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-08-30T19:00:48.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:00:48.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:00:48.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa76d950-f2c4-4740-8ac4-578c516bd4b1 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aa76d950-f2c4-4740-8ac4-578c516bd4b1' 2024-08-30T19:00:48.445 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 aa76d950-f2c4-4740-8ac4-578c516bd4b1 2024-08-30T19:00:48.446 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:00:48.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgFtJmh4yKGxAA+ISo//Gl15iOQ6L+Inf71g== 2024-08-30T19:00:48.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgFtJmh4yKGxAA+ISo//Gl15iOQ6L+Inf71g=="}' 2024-08-30T19:00:48.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa76d950-f2c4-4740-8ac4-578c516bd4b1 -i td/bad-inc-map/0/new.json 2024-08-30T19:00:48.727 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:00:48.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-08-30T19:00:48.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgFtJmh4yKGxAA+ISo//Gl15iOQ6L+Inf71g== --osd-uuid aa76d950-f2c4-4740-8ac4-578c516bd4b1 2024-08-30T19:00:48.825 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:48.824+0000 7f5110b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:48.828 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:48.828+0000 7f5110b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:48.829 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:48.828+0000 7f5110b2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:48.830 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:48.828+0000 7f5110b2a7c0 -1 bdev(0x55fe38748a80 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:00:48.830 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:48.828+0000 7f5110b2a7c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-08-30T19:00:54.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-08-30T19:00:54.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:00:54.702 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:00:54.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:00:54.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:00:54.949 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:00:54.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:00:54.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:00:54.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:00:54.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:00:54.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:00:54.981 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:54.980+0000 7fb0a5c007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:54.983 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:54.984+0000 7fb0a5c007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:54.984 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:54.984+0000 7fb0a5c007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:00:55.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:00:55.173 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:00:55.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:00:55.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:00:55.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:00:55.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:00:56.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:00:56.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:00:56.404 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:00:56.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:00:56.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:00:56.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:00:56.544 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:56.544+0000 7fb0a5c007c0 -1 Falling back to public interface 2024-08-30T19:00:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:00:57.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:00:57.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:00:57.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:00:57.695 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:00:57.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:00:57.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:00:58.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:00:59.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:00:59.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:00:59.120 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:00:59.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:00:59.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:00:59.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:00:59.177 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:00:59.176+0000 7fb0a5c007c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:00:59.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:00.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:00.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:00.717 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:01:00.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:01:00.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:00.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:01.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:02.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:02.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:02.198 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:01:02.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:01:02.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:02.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:02.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:03.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:03.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:03.615 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:01:03.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:01:03.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:03.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:04.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:05.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:05.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:05.183 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:01:05.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:01:05.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:05.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:05.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:06.602 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:06.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:06.603 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:01:06.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:01:06.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:06.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:07.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:08.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:08.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:08.177 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:01:08.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:01:08.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:08.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:08.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:09.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:09.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:09.624 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:01:09.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:01:09.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:09.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:10.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:11.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:11.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:11.241 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:01:11.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:01:11.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:11.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:11.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:12.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:12.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:12.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:01:12.736 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:01:12.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:12.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:13.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:14.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:14.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:14.451 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:01:14.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:01:14.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:14.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:15.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:16.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:16.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:16.070 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:01:16.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:01:16.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:16.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:16.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:17.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:17.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:17.914 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:01:17.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:01:17.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:17.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:18.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:19.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:19.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:19.642 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:01:19.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:01:19.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:19.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:20.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:21.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:21.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:21.677 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:01:21.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:01:21.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:21.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:22.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:23.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:23.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:23.709 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:01:23.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:01:23.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:23.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:01:24.214 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/344089365,v1:127.0.0.1:6803/344089365] [v2:127.0.0.1:6804/344089365,v1:127.0.0.1:6805/344089365] exists,up aa76d950-f2c4-4740-8ac4-578c516bd4b1 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-08-30T19:01:24.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:01:24.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:01:24.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-08-30T19:01:24.221 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:01:24.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ca5c0d8-763d-44e8-85fd-e5a31d786011 2024-08-30T19:01:24.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3ca5c0d8-763d-44e8-85fd-e5a31d786011' 2024-08-30T19:01:24.221 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 3ca5c0d8-763d-44e8-85fd-e5a31d786011 2024-08-30T19:01:24.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:01:24.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEF9JmdXMQDhAAvzBBESCQmPbdC22UE4EGkA== 2024-08-30T19:01:24.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEF9JmdXMQDhAAvzBBESCQmPbdC22UE4EGkA=="}' 2024-08-30T19:01:24.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ca5c0d8-763d-44e8-85fd-e5a31d786011 -i td/bad-inc-map/1/new.json 2024-08-30T19:01:24.754 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:01:24.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-08-30T19:01:24.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEF9JmdXMQDhAAvzBBESCQmPbdC22UE4EGkA== --osd-uuid 3ca5c0d8-763d-44e8-85fd-e5a31d786011 2024-08-30T19:01:24.801 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:24.799+0000 7fe66a5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:24.809 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:24.807+0000 7fe66a5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:24.810 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:24.811+0000 7fe66a5b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:24.811 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:24.811+0000 7fe66a5b57c0 -1 bdev(0x5648dd8e8a80 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:01:24.811 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:24.811+0000 7fe66a5b57c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-08-30T19:01:30.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-08-30T19:01:30.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:01:30.703 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:01:30.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:01:30.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:01:31.252 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:01:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:01:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:01:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:01:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:01:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:01:31.281 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:31.279+0000 7f441d7bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:31.283 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:31.283+0000 7f441d7bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:31.284 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:31.283+0000 7f441d7bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:01:31.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:31.708 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:01:31.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:01:31.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:31.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:32.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:32.765 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:32.763+0000 7f441d7bd7c0 -1 Falling back to public interface 2024-08-30T19:01:33.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:33.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:33.167 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:01:33.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:01:33.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:33.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:33.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:34.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:34.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:34.591 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:01:34.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:01:34.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:34.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:34.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:35.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:01:35.627+0000 7f441d7bd7c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:01:35.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:35.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:35.990 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:01:35.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:01:35.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:35.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:36.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:37.451 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:01:37.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:01:37.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:37.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:37.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:38.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:38.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:38.855 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:01:38.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:01:38.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:38.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:39.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:40.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:40.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:40.460 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:01:40.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:01:40.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:40.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:41.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:42.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:42.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:42.018 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:01:42.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:01:42.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:42.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:42.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:43.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:43.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:01:43.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:01:43.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:43.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:43.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:44.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:44.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:44.929 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:01:44.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:01:44.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:44.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:45.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:46.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:46.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:46.419 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:01:46.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:01:46.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:46.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:46.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:47.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:47.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:47.921 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:01:47.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:01:47.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:47.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:48.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:49.703 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:01:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:01:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:50.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:51.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:51.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:51.461 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:01:51.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:01:51.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:51.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:52.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:53.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:53.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:53.294 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:01:53.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:01:53.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:53.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:54.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:55.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:55.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:55.304 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:01:55.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:01:55.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:55.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:56.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:57.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:57.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:57.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:01:57.459 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:01:57.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:57.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:58.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:01:59.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:01:59.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:01:59.171 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:01:59.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:01:59.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:01:59.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:01:59.648 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/112980349,v1:127.0.0.1:6811/112980349] [v2:127.0.0.1:6812/112980349,v1:127.0.0.1:6813/112980349] exists,up 3ca5c0d8-763d-44e8-85fd-e5a31d786011 2024-08-30T19:01:59.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:01:59.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:01:59.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-08-30T19:01:59.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:01:59.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:01:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-08-30T19:01:59.660 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:01:59.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4b3a92f-1fdb-4617-9635-72382b4ae2c2 2024-08-30T19:01:59.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d4b3a92f-1fdb-4617-9635-72382b4ae2c2' 2024-08-30T19:01:59.661 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 d4b3a92f-1fdb-4617-9635-72382b4ae2c2 2024-08-30T19:01:59.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:01:59.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnF9Jmo9VUKBAAfjQAklstGKX5n56ub5A+7A== 2024-08-30T19:01:59.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnF9Jmo9VUKBAAfjQAklstGKX5n56ub5A+7A=="}' 2024-08-30T19:01:59.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4b3a92f-1fdb-4617-9635-72382b4ae2c2 -i td/bad-inc-map/2/new.json 2024-08-30T19:02:00.211 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:02:00.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-08-30T19:02:00.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnF9Jmo9VUKBAAfjQAklstGKX5n56ub5A+7A== --osd-uuid d4b3a92f-1fdb-4617-9635-72382b4ae2c2 2024-08-30T19:02:00.253 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:00.250+0000 7f69649217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:00.271 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:00.266+0000 7f69649217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:00.272 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:00.270+0000 7f69649217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:00.272 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:00.270+0000 7f69649217c0 -1 bdev(0x55fa11958a80 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:02:00.273 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:00.270+0000 7f69649217c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-08-30T19:02:06.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-08-30T19:02:06.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:02:06.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:02:06.106 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:02:06.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:02:06.661 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:02:06.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:02:06.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:02:06.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5704d489-96f9-4e63-9f4d-55208f609db0 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21391/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:02:06.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:02:06.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:02:06.691 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:06.685+0000 7f18c98607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:06.724 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:06.721+0000 7f18c98607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:06.725 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:06.721+0000 7f18c98607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:07.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:02:07.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:02:07.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:02:07.074 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:02:07.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:02:07.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:02:07.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:07.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:02:07.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:07.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:07.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:08.238 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:08.233+0000 7f18c98607c0 -1 Falling back to public interface 2024-08-30T19:02:08.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:08.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:08.486 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:02:08.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:02:08.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:08.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:08.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:09.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:09.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:09.935 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:02:09.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:02:09.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:09.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:10.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:11.267 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:11.261+0000 7f18c98607c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:02:11.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:11.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:11.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:02:11.346 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:02:11.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:11.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:11.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:12.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:12.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:12.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:02:12.948 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:02:12.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:12.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:13.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:14.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:14.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:14.438 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:02:14.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:02:14.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:14.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:14.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:16.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:16.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:16.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:02:16.001 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:02:16.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:16.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:16.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:17.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:17.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:17.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:02:17.439 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:02:17.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:17.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:17.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:18.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:18.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:18.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:02:18.874 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:02:18.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:18.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:19.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:20.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:20.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:20.444 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:02:20.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:02:20.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:20.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:20.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:21.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:21.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:21.921 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:02:21.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:02:21.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:21.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:22.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:23.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:23.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:23.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:02:23.386 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:02:23.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:23.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:24.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:25.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:25.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:25.513 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:02:25.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:02:25.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:25.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:26.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:27.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:27.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:27.257 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:02:27.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:02:27.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:27.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:28.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:29.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:29.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:29.120 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:02:29.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:02:29.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:29.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:30.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:31.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:31.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:31.061 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:02:31.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:02:31.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:31.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:31.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:32.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:32.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:32.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:02:32.821 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:02:32.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:32.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:33.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:02:34.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:02:34.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:02:34.265 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:02:34.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:02:34.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:34.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:34.693 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3813541931,v1:127.0.0.1:6819/3813541931] [v2:127.0.0.1:6820/3813541931,v1:127.0.0.1:6821/3813541931] exists,up d4b3a92f-1fdb-4617-9635-72382b4ae2c2 2024-08-30T19:02:34.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:02:34.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:02:34.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:02:34.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-08-30T19:02:35.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-08-30T19:02:35.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-08-30T19:02:35.817 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-08-30T19:02:35.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:02:36.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-08-30T19:02:37.764 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 1 2024-08-30T19:02:37.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-08-30T19:02:38.968 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-08-30T19:02:38.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-08-30T19:02:43.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-08-30T19:02:43.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-30T19:02:43.992 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:02:43.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:02:43.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:43.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:02:44.407 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/344089365,v1:127.0.0.1:6803/344089365] [v2:127.0.0.1:6804/344089365,v1:127.0.0.1:6805/344089365] exists,up aa76d950-f2c4-4740-8ac4-578c516bd4b1 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:02:44.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:44.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:02:44.823 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/112980349,v1:127.0.0.1:6811/112980349] [v2:127.0.0.1:6812/112980349,v1:127.0.0.1:6813/112980349] exists,up 3ca5c0d8-763d-44e8-85fd-e5a31d786011 2024-08-30T19:02:44.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:02:44.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:02:44.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-30T19:02:44.825 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:02:44.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:02:44.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:02:44.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:02:45.295 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3813541931,v1:127.0.0.1:6819/3813541931] [v2:127.0.0.1:6820/3813541931,v1:127.0.0.1:6821/3813541931] exists,up d4b3a92f-1fdb-4617-9635-72382b4ae2c2 2024-08-30T19:02:45.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:02:45.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:02:45.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:02:45.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-08-30T19:02:45.298 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:35.661+0000 7f18ba804640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.298 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:35.665+0000 7f18b8800640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.298 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:36.165+0000 7f18ba804640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:36.441+0000 7f18ba804640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.365+0000 7f18b8800640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.557+0000 7f18b8800640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.657+0000 7f18b8800640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.965+0000 7f18b8800640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.085+0000 7f18b8800640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.249+0000 7f18ba804640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.605+0000 7f18ba804640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.033+0000 7f18ba804640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.393+0000 7f18ba804640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.393+0000 7f18ba804640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.299 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:40.156+0000 7f18ba804640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-08-30T19:02:45.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:35.661+0000 7f18ba804640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:35.669+0000 7f18b8800640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:36.169+0000 7f18ba804640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:36.441+0000 7f18ba804640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.369+0000 7f18b8800640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.557+0000 7f18b8800640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.657+0000 7f18b8800640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:37.965+0000 7f18b8800640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.085+0000 7f18b8800640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.253+0000 7f18ba804640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:38.605+0000 7f18ba804640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.033+0000 7f18ba804640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-08-30T19:02:45.302 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.393+0000 7f18ba804640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:39.397+0000 7f18ba804640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:02:40.156+0000 7f18ba804640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-08-30T19:02:45.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-08-30T19:02:45.824 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' does not exist 2024-08-30T19:02:45.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-08-30T19:02:45.841 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:02:45.841 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:02:45.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:02:45.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:02:45.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:02:51.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:02:51.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-08-30T19:02:51.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-30T19:02:51.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:02:51.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-30T19:02:51.159 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:02:51.159 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:02:51.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:02:51.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:02:51.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:02:51.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:02:51.166 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:02:51.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:02:51.167 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:02:51.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:02:51.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:02:51.169 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:02:51.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:02:51.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:02:51.170 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:02:51.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:02:51.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-30T19:02:51.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:02:51.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:02:51.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21391 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/bad-inc-map 0 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-30T19:02:51.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-30T19:02:51.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:02:51.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:02:51.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:02:51.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:02:51.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:02:51.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:02:51.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:02:51.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:02:51.192 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:02:51.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:02:51.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:02:51.193 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:02:51.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:02:51.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:02:51.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:02:51.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-30T19:02:51.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-30T19:02:51.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:02:51.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21391 2024-08-30T19:02:51.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21391 2024-08-30T19:02:51.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:02:51.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:02:51.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-30T19:02:51.200 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-30T19:02:51.201 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-30T19:02:51.258 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-08-30T19:02:51.258 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-08-30T19:02:51.311 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-30T19:02:51.317 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-30T19:02:51.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/divergent-priors 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-08-30T19:02:51.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-08-30T19:02:51.319 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-08-30T19:02:51.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-08-30T19:02:51.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-08-30T19:02:51.319 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-08-30T19:02:51.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-08-30T19:02:51.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-08-30T19:02:51.320 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-08-30T19:02:51.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-08-30T19:02:51.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-08-30T19:02:51.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-08-30T19:02:51.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-08-30T19:02:51.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-08-30T19:02:51.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-08-30T19:02:51.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-08-30T19:02:51.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-08-30T19:02:51.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-08-30T19:02:51.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-08-30T19:02:51.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-08-30T19:02:51.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-08-30T19:02:51.323 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none ' 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-08-30T19:02:51.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_2 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_3 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_ec' 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:02:51.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:02:51.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:02:51.330 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:02:51.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:02:51.331 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:02:51.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:02:51.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:02:51.333 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:02:51.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:02:51.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:02:51.334 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:02:51.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:02:51.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:02:51.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:02:51.337 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:02:51.337 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.338 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:51.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:02:51.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:02:51.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:02:51.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-30T19:02:51.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:02:51.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:51.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27261 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-08-30T19:02:51.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-08-30T19:02:51.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-08-30T19:02:51.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-08-30T19:02:51.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-30T19:02:51.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-30T19:02:51.755 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:02:51.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:02:51.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:02:51.756 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:02:51.756 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.756 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:51.757 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:02:51.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:02:51.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:02:51.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:02:51.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:02:51.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:02:51.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:51.967 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:02:51.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:02:51.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get fsid 2024-08-30T19:02:52.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:02:52.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:02:52.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:02:52.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:52.084 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:02:52.085 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:02:52.085 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get mon_host 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-30T19:02:52.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:02:53.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:02:53.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:02:53.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:02:53.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:02:53.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-30T19:02:53.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-08-30T19:02:53.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:02:53.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:02:53.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:02:53.243 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:02:53.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:02:53.245 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:02:53.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a2fdc2c8-75f3-4a0f-b764-fdf422f32a14' 2024-08-30T19:02:53.246 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:02:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdF9JmOmPhDxAAsM83B4+0iZMmhJKOVGDVxA== 2024-08-30T19:02:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdF9JmOmPhDxAAsM83B4+0iZMmhJKOVGDVxA=="}' 2024-08-30T19:02:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 -i td/divergent-priors/0/new.json 2024-08-30T19:02:53.553 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:02:53.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-30T19:02:53.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdF9JmOmPhDxAAsM83B4+0iZMmhJKOVGDVxA== --osd-uuid a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:02:53.610 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:53.608+0000 7fabd65f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:53.621 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:53.620+0000 7fabd65f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:53.623 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:53.620+0000 7fabd65f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:02:53.624 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:53.624+0000 7fabd65f87c0 -1 bdev(0x562ea36dca80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:02:53.624 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:02:53.624+0000 7fabd65f87c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-30T19:02:59.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-30T19:02:59.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:02:59.552 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:02:59.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:02:59.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:03:00.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:03:00.122 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:03:00.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:03:00.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:03:00.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:03:00.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:03:00.162 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:00.160+0000 7fc73717e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:00.164 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:00.164+0000 7fc73717e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:00.165 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:00.164+0000 7fc73717e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:00.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:03:00.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:03:00.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:03:00.540 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:03:00.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:03:00.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:03:00.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:00.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:03:00.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:00.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:00.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:01.178 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:01.176+0000 7fc73717e7c0 -1 Falling back to public interface 2024-08-30T19:03:01.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:01.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:01.951 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:03:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:03:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:02.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:03.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:03.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:03.418 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:03:03.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:03:03.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:03.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:03.773 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:03.772+0000 7fc73717e7c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:03:03.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:04.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:04.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:04.861 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:03:04.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:03:04.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:04.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:05.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:06.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:06.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:06.296 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:03:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:03:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:06.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:07.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:07.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:07.872 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:03:07.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:03:07.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:07.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:08.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:09.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:09.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:09.300 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:03:09.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:03:09.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:09.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:09.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:10.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:10.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:10.765 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:03:10.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:03:10.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:10.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:11.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:12.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:12.207 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:03:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:03:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:12.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:13.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:13.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:13.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:03:13.704 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:03:13.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:13.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:14.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:15.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:15.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:15.169 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:03:15.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:03:15.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:15.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:15.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:16.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:16.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:16.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:03:16.597 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:03:16.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:16.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:17.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:18.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:18.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:18.741 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:03:18.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:03:18.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:18.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:19.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:20.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:20.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:20.702 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:03:20.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:03:20.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:20.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:21.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:22.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:22.279 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:03:22.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:22.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:03:22.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:22.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:23.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:24.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:24.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:24.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:03:24.452 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:03:24.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:24.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:25.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:26.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:26.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:26.483 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:03:26.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:03:26.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:26.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:03:26.993 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/553511242,v1:127.0.0.1:6803/553511242] [v2:127.0.0.1:6804/553511242,v1:127.0.0.1:6805/553511242] exists,up a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:03:26.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:03:26.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:03:26.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:03:26.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:03:26.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:03:26.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:03:26.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:03:26.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:03:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:03:27.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:03:27.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:03:27.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:03:27.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 112bf508-eac0-4def-a4cd-b016ce41f642' 2024-08-30T19:03:27.002 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:03:27.003 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:03:27.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/F9Jm9mzNARAAhFB4zEJYoKTP49FNY+ykBQ== 2024-08-30T19:03:27.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/F9Jm9mzNARAAhFB4zEJYoKTP49FNY+ykBQ=="}' 2024-08-30T19:03:27.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 112bf508-eac0-4def-a4cd-b016ce41f642 -i td/divergent-priors/1/new.json 2024-08-30T19:03:27.552 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:03:27.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-30T19:03:27.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/F9Jm9mzNARAAhFB4zEJYoKTP49FNY+ykBQ== --osd-uuid 112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:03:27.612 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:27.611+0000 7f68bcfe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:27.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:27.623+0000 7f68bcfe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:27.630 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:27.623+0000 7f68bcfe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:27.630 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:27.623+0000 7f68bcfe97c0 -1 bdev(0x561687ff4a80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:03:27.630 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:27.623+0000 7f68bcfe97c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-30T19:03:33.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-30T19:03:33.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:03:33.504 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:03:33.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:03:33.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:03:34.061 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:03:34.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:03:34.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:03:34.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:03:34.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:03:34.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:03:34.102 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:34.099+0000 7f1daec927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:34.121 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:34.119+0000 7f1daec927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:34.122 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:34.119+0000 7f1daec927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:03:34.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:03:34.505 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:03:34.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:34.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:34.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:35.190 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:35.187+0000 7f1daec927c0 -1 Falling back to public interface 2024-08-30T19:03:35.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:35.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:35.911 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:03:35.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:03:35.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:35.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:36.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:37.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:37.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:37.360 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:03:37.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:03:37.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:37.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:37.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:03:37.931+0000 7f1daec927c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:03:38.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:38.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:38.805 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:03:38.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:03:38.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:38.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:39.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:40.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:40.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:40.321 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:03:40.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:03:40.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:40.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:40.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:41.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:41.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:41.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:03:41.993 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:03:41.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:41.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:42.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:43.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:43.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:43.611 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:03:43.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:03:43.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:43.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:44.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:45.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:45.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:45.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:03:45.141 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:03:45.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:45.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:45.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:46.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:47.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:47.544 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:03:47.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:03:47.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:47.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:47.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:48.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:48.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:48.117 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:03:48.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:03:48.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:48.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:48.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:49.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:49.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:49.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:03:49.601 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:03:49.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:49.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:50.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:51.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:51.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:51.455 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:03:51.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:03:51.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:51.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:52.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:53.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:53.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:53.474 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:03:53.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:03:53.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:53.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:54.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:55.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:55.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:55.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:03:55.024 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:03:55.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:55.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:55.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:56.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:56.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:56.736 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:03:56.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:03:56.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:56.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:57.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:03:58.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:03:58.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:03:58.729 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:03:58.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:03:58.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:03:58.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:03:59.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:00.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:00.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:00.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:04:00.510 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:04:00.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:00.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:04:01.070 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2976352712,v1:127.0.0.1:6811/2976352712] [v2:127.0.0.1:6812/2976352712,v1:127.0.0.1:6813/2976352712] exists,up 112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:04:01.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:04:01.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:04:01.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:04:01.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:04:01.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:04:01.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:04:01.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:04:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:04:01.100 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:04:01.101 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:04:01.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:04:01.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d6e2cee9-0807-4805-a52a-83cda77e491b' 2024-08-30T19:04:01.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:04:01.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChF9JmbnHVBxAAiPxOHRMqFXNxhKTUOJct8A== 2024-08-30T19:04:01.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChF9JmbnHVBxAAiPxOHRMqFXNxhKTUOJct8A=="}' 2024-08-30T19:04:01.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6e2cee9-0807-4805-a52a-83cda77e491b -i td/divergent-priors/2/new.json 2024-08-30T19:04:01.614 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:04:01.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-30T19:04:01.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChF9JmbnHVBxAAiPxOHRMqFXNxhKTUOJct8A== --osd-uuid d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:04:01.672 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:01.666+0000 7f30d92bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:01.673 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:01.670+0000 7f30d92bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:01.675 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:01.670+0000 7f30d92bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:01.675 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:01.670+0000 7f30d92bd7c0 -1 bdev(0x55d32e48ca80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:04:01.675 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:01.670+0000 7f30d92bd7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-30T19:04:07.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-30T19:04:07.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:04:07.663 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:04:07.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:04:07.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:04:08.240 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:04:08.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:04:08.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:04:08.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:04:08.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:04:08.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:04:08.280 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:08.278+0000 7fbb53b3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:08.283 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:08.282+0000 7fbb53b3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:08.285 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:08.282+0000 7fbb53b3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:04:08.671 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:04:08.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:08.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:09.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:09.354 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:09.350+0000 7fbb53b3e7c0 -1 Falling back to public interface 2024-08-30T19:04:10.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:10.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:10.116 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:04:10.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:04:10.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:10.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:10.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:04:11.597 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:04:11.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:11.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:12.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:12.060 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:04:12.058+0000 7fbb53b3e7c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:04:13.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:13.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:13.041 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:04:13.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:04:13.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:13.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:13.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:14.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:14.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:14.479 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:04:14.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:04:14.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:14.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:15.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:16.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:16.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:16.048 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:04:16.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:04:16.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:16.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:16.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:17.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:17.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:17.590 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:04:17.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:04:17.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:17.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:17.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:18.985 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:04:18.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:04:18.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:18.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:19.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:20.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:20.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:20.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:04:20.453 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:04:20.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:20.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:20.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:21.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:21.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:21.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:04:21.996 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:04:21.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:21.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:22.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:23.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:23.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:23.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:04:23.557 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:04:23.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:23.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:24.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:25.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:25.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:25.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:04:25.457 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:04:25.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:25.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:26.278 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:27.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:27.281 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:04:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:04:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:27.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:28.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:29.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:29.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:29.027 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:04:29.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:04:29.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:29.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:30.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:31.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:31.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:31.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:04:31.093 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:04:31.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:31.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:31.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:32.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:04:32.876 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:04:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:33.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:04:34.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:04:34.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:04:34.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:04:34.785 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:04:34.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:04:34.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:04:35.203 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3456051421,v1:127.0.0.1:6819/3456051421] [v2:127.0.0.1:6820/3456051421,v1:127.0.0.1:6821/3456051421] exists,up d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:04:35.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:04:35.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:04:35.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:04:35.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-08-30T19:04:35.852 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-08-30T19:04:35.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-08-30T19:04:36.546 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-08-30T19:04:36.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-08-30T19:04:37.358 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-08-30T19:04:37.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-08-30T19:04:37.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-30T19:04:38.027 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-08-30T19:04:38.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:04:39.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-08-30T19:04:39.800 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-08-30T19:04:39.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-08-30T19:04:40.690 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-08-30T19:04:40.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-08-30T19:04:40.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:04:40.713 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:04:41.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:04:41.202 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:04:41.202 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:04:41.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:04:41.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:41.203 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:04:41.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-08-30T19:04:41.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-08-30T19:04:41.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-08-30T19:04:41.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:41.373 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:04:41.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-08-30T19:04:41.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-08-30T19:04:41.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973' 2024-08-30T19:04:41.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:41.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:04:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-08-30T19:04:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-08-30T19:04:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973 2-64424509450' 2024-08-30T19:04:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:41.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-08-30T19:04:41.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:41.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:04:41.696 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-08-30T19:04:41.696 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:41.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-08-30T19:04:41.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-30T19:04:41.698 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836503 2024-08-30T19:04:41.698 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:04:42.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-08-30T19:04:42.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:42.163 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-08-30T19:04:42.163 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:42.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:04:42.166 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-08-30T19:04:42.166 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:42.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-08-30T19:04:42.167 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672973 2024-08-30T19:04:42.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-30T19:04:42.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:04:42.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-30T19:04:42.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:42.598 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-08-30T19:04:42.599 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:42.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:04:42.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-08-30T19:04:42.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:42.603 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509450 2024-08-30T19:04:42.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-08-30T19:04:42.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-30T19:04:42.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:04:43.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-08-30T19:04:43.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-08-30T19:04:43.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:04:43.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:04:43.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:04:43.077 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:04:43.078 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:04:43.078 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:04:43.078 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:04:43.078 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:04:43.078 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:04:43.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:04:43.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:04:43.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:04:43.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:04:43.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:04:43.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:04:43.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:04:43.580 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:04:43.580 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:04:43.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:04:43.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:43.580 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:04:43.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-30T19:04:43.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-30T19:04:43.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-30T19:04:43.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:43.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:04:43.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-08-30T19:04:43.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-08-30T19:04:43.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975' 2024-08-30T19:04:43.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:04:43.902 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:04:44.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-08-30T19:04:44.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-08-30T19:04:44.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975 2-64424509452' 2024-08-30T19:04:44.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:44.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-30T19:04:44.067 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:44.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:04:44.068 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:44.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-30T19:04:44.069 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836505 2024-08-30T19:04:44.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-30T19:04:44.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-30T19:04:44.070 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:04:44.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-30T19:04:44.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:44.582 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-08-30T19:04:44.582 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:44.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:04:44.585 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-08-30T19:04:44.585 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:44.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-08-30T19:04:44.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-08-30T19:04:44.586 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672975 2024-08-30T19:04:44.587 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:04:45.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672975 2024-08-30T19:04:45.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:04:46.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:04:46.057 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:04:46.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-08-30T19:04:46.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:04:46.500 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-08-30T19:04:46.500 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:04:46.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:04:46.502 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-08-30T19:04:46.502 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:04:46.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-08-30T19:04:46.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-08-30T19:04:46.504 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509452 2024-08-30T19:04:46.505 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:04:46.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-08-30T19:04:46.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:04:46.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:04:46.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:04:47.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:04:47.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:04:47.553 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:04:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:04:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:04:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:04:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:04:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:04:47.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:04:47.971 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:04:47.972 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:04:47.972 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:04:48.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:04:48.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:04:48.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:04:48.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-08-30T19:04:48.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-08-30T19:04:48.915 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:04:48.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-08-30T19:04:48.931 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-08-30T19:04:48.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-08-30T19:04:48.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-08-30T19:04:49.331 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:04:49.331 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:04:39.826470+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:08:07.321938+0000 0 0 2024-08-30T19:04:49.331 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:04:49.331 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:04:49.332 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:04:49.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-08-30T19:04:49.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-30T19:04:49.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-08-30T19:04:49.348 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-08-30T19:04:49.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-08-30T19:04:49.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-08-30T19:04:49.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:49.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-08-30T19:04:49.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:49.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-08-30T19:04:49.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:49.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-08-30T19:04:49.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:49.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-08-30T19:04:49.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:49.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-08-30T19:04:50.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-08-30T19:04:50.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-08-30T19:04:50.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-08-30T19:04:50.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-08-30T19:04:50.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-08-30T19:04:50.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:50.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-08-30T19:04:51.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-08-30T19:04:51.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-08-30T19:04:51.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-08-30T19:04:51.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-08-30T19:04:51.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-08-30T19:04:51.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:51.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-08-30T19:04:52.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-08-30T19:04:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-08-30T19:04:52.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-08-30T19:04:52.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-08-30T19:04:52.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-08-30T19:04:52.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:52.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-08-30T19:04:53.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-08-30T19:04:53.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-08-30T19:04:53.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-08-30T19:04:53.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-08-30T19:04:53.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-08-30T19:04:53.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-08-30T19:04:53.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:53.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-08-30T19:04:54.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-08-30T19:04:54.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-08-30T19:04:54.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-08-30T19:04:54.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-08-30T19:04:54.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-08-30T19:04:54.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:54.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-08-30T19:04:55.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-08-30T19:04:55.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-08-30T19:04:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-08-30T19:04:55.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-08-30T19:04:55.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-08-30T19:04:55.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:55.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-08-30T19:04:56.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-08-30T19:04:56.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-08-30T19:04:56.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-08-30T19:04:56.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-08-30T19:04:56.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-08-30T19:04:56.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:56.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-08-30T19:04:57.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-08-30T19:04:57.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-08-30T19:04:57.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-08-30T19:04:57.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-08-30T19:04:57.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-08-30T19:04:57.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-08-30T19:04:57.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:57.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-08-30T19:04:58.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-08-30T19:04:58.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-08-30T19:04:58.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-08-30T19:04:58.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-08-30T19:04:58.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-08-30T19:04:58.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:58.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-08-30T19:04:59.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:59.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-08-30T19:04:59.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:59.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-08-30T19:04:59.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:59.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-08-30T19:04:59.756 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:59.756 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-08-30T19:04:59.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:04:59.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-08-30T19:05:00.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-08-30T19:05:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-08-30T19:05:00.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-08-30T19:05:00.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-08-30T19:05:00.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-08-30T19:05:00.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:00.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-08-30T19:05:01.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-08-30T19:05:01.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-08-30T19:05:01.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-08-30T19:05:01.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-08-30T19:05:01.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-08-30T19:05:01.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-08-30T19:05:01.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:01.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-08-30T19:05:02.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-08-30T19:05:02.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-08-30T19:05:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-08-30T19:05:02.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-08-30T19:05:02.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-08-30T19:05:02.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:02.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-08-30T19:05:03.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-08-30T19:05:03.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-08-30T19:05:03.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-08-30T19:05:03.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-08-30T19:05:03.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-08-30T19:05:03.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:03.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-08-30T19:05:04.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-08-30T19:05:04.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-08-30T19:05:04.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-08-30T19:05:04.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-08-30T19:05:04.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-08-30T19:05:04.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:04.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-08-30T19:05:05.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:05.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-08-30T19:05:05.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:05.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-08-30T19:05:05.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-30T19:05:05.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-08-30T19:05:05.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:05:05.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-08-30T19:05:05.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:05:05.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:05:05.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:05:05.571 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:05:05.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:05:05.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:05:05.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:05:05.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:05:05.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:05:05.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:05:06.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:05:06.130 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:05:06.130 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:05:06.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:05:06.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:06.131 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:05:06.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-08-30T19:05:06.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-08-30T19:05:06.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-08-30T19:05:06.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:06.305 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:05:06.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-08-30T19:05:06.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-08-30T19:05:06.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672981' 2024-08-30T19:05:06.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:06.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:05:06.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-08-30T19:05:06.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-08-30T19:05:06.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672981 2-64424509459' 2024-08-30T19:05:06.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:05:06.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-08-30T19:05:06.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:05:06.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:05:06.634 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-08-30T19:05:06.634 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:05:06.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-08-30T19:05:06.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-08-30T19:05:06.636 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836512 2024-08-30T19:05:06.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:05:07.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836512 2024-08-30T19:05:07.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:05:08.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:05:08.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:05:08.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-08-30T19:05:08.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:05:08.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-08-30T19:05:08.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:05:08.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:05:08.522 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-08-30T19:05:08.522 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:05:08.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-08-30T19:05:08.523 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672981 2024-08-30T19:05:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-08-30T19:05:08.524 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:05:08.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672981 2024-08-30T19:05:08.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:05:09.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-08-30T19:05:09.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:05:09.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:05:09.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-08-30T19:05:09.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:05:09.003 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509459 2024-08-30T19:05:09.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-08-30T19:05:09.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-30T19:05:09.003 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:05:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-30T19:05:09.477 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:05:09.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:09.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:10.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:05:10.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:10.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:10.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:05:10.412 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:10.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:10.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:10.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:05:10.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:05:10.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:05:10.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-08-30T19:05:10.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-30T19:05:10.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-30T19:05:10.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-30T19:05:10.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-30T19:05:11.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-08-30T19:05:11.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-08-30T19:05:11.421 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-08-30T19:05:11.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-08-30T19:05:11.842 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:05:11.842 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-30T19:04:39.826470+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:08:07.321938+0000 0 0 2024-08-30T19:05:11.842 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:05:11.842 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:05:11.842 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:05:11.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:11.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-08-30T19:05:11.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:05:11.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:05:11.861 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:05:11.861 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:11.862 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:11.862 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:05:11.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-08-30T19:05:11.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-30T19:05:11.966 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:05:11.967 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:05:11.967 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:05:11.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:11.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:05:11.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-08-30T19:05:11.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-30T19:05:12.092 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:05:12.092 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:05:12.092 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:05:12.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-08-30T19:05:12.106 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-08-30T19:05:12.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-08-30T19:05:12.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-08-30T19:05:12.108 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent object 2024-08-30T19:05:12.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-08-30T19:05:12.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-08-30T19:05:12.109 INFO:tasks.workunit.client.0.smithi079.stdout:create missing divergent object 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=test 2024-08-30T19:05:12.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=existing_99 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/divergent-priors 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds test existing_99 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-08-30T19:05:12.111 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-08-30T19:05:12.112 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-08-30T19:05:12.112 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:0 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0' '2') 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-08-30T19:05:12.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/divergent-priors/1/type 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-30T19:05:12.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:05:12.577 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27261/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-30T19:05:12.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-30T19:05:12.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-08-30T19:05:12.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-08-30T19:05:12.700 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-08-30T19:05:12.700 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:05:12.700 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:05:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:05:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:12.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:05:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-08-30T19:05:12.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok injectdataerr test existing_99 2024-08-30T19:05:12.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-08-30T19:05:12.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-08-30T19:05:22.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-08-30T19:05:22.831 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-08-30T19:05:22.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-08-30T19:05:22.832 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-08-30T19:05:22.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-08-30T19:05:23.242 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:05:23.242 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-08-30T19:05:12.970992+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:08:07.321938+0000 0 0 2024-08-30T19:05:23.242 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:05:23.242 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:05:23.242 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:05:23.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-08-30T19:05:23.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:05:23.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:05:23.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:05:23.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:05:23.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:05:23.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:05:23.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-30T19:05:23.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-08-30T19:05:24.229 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:05:24.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-30T19:05:24.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-08-30T19:05:24.813 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:05:24.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-30T19:05:24.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-08-30T19:05:25.370 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:05:25.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:25.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-08-30T19:05:25.908 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-08-30T19:05:25.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:25.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-08-30T19:05:26.450 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-08-30T19:05:26.466 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-08-30T19:05:26.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-08-30T19:05:26.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-08-30T19:05:26.859 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:05:26.859 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-08-30T19:05:12.970992+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:08:07.321938+0000 0 0 2024-08-30T19:05:26.859 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:05:26.859 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:05:26.859 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:05:26.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:26.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-08-30T19:05:26.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:05:26.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:05:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:05:26.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:05:26.881 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:05:26.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:05:26.917 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:26.913+0000 7f217b6817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:26.919 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:26.917+0000 7f217b6817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:26.921 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:26.917+0000 7f217b6817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:27.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:05:27.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:05:27.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:05:27.345 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:05:27.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:05:27.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:05:27.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:27.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:05:27.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:27.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:05:27.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:27.974 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:27.969+0000 7f217b6817c0 -1 Falling back to public interface 2024-08-30T19:05:28.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:28.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:28.861 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:05:28.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:05:28.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:28.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:05:29.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:30.278 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:30.278 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:30.278 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:05:30.279 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:05:30.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:30.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:05:30.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:31.348+0000 7f217b6817c0 -1 osd.0 29 log_to_monitors true 2024-08-30T19:05:31.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:31.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:31.980 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:05:31.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:05:31.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:31.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:05:32.438 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2553705694,v1:127.0.0.1:6803/2553705694] [v2:127.0.0.1:6804/2553705694,v1:127.0.0.1:6805/2553705694] exists,up a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:05:32.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:05:32.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:05:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:05:32.443 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:05:32.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:05:32.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:05:32.444 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-30T19:05:32.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:05:32.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:05:32.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:05:32.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:05:32.474 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:32.468+0000 7f3cd85347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:32.537 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:32.532+0000 7f3cd85347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:32.538 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:32.532+0000 7f3cd85347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:05:32.959 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:05:32.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:32.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:05:33.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:34.066 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:34.060+0000 7f3cd85347c0 -1 Falling back to public interface 2024-08-30T19:05:34.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:34.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:34.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:05:34.402 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:05:34.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:34.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:05:34.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:35.817 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:05:35.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:35.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:35.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:05:35.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:35.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:05:36.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:05:37.061 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:05:37.056+0000 7f3cd85347c0 -1 osd.2 29 log_to_monitors true 2024-08-30T19:05:37.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:05:37.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:05:37.246 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:05:37.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:05:37.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:05:37.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:05:37.699 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/4270653665,v1:127.0.0.1:6811/4270653665] [v2:127.0.0.1:6812/4270653665,v1:127.0.0.1:6813/4270653665] exists,up d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:05:37.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:05:37.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:05:37.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:05:37.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:37.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-08-30T19:05:38.246 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-08-30T19:05:38.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-08-30T19:05:38.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-08-30T19:05:39.054 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-08-30T19:05:39.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:05:39.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-08-30T19:05:39.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:05:39.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:05:39.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:05:39.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:05:39.076 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:05:39.076 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:05:39.077 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:05:39.077 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:05:39.077 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:05:39.147 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:39.607 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:05:39.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822660 2024-08-30T19:05:39.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822660 2024-08-30T19:05:39.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822660' 2024-08-30T19:05:39.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:39.766 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:05:39.887 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T19:05:39.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:05:39.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:05:39.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:05:39.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:05:39.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:05:40.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757251 2024-08-30T19:05:40.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757251 2024-08-30T19:05:40.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-08-30T19:05:40.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:05:40.050 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-154618822660 2024-08-30T19:05:40.050 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:05:40.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:05:40.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-154618822660 2024-08-30T19:05:40.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:05:40.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822660 2024-08-30T19:05:40.052 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 154618822660 2024-08-30T19:05:40.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-08-30T19:05:40.052 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:05:40.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822660 -lt 154618822660 2024-08-30T19:05:40.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:05:40.471 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:05:40.471 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-163208757251 2024-08-30T19:05:40.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:05:40.473 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-163208757251 2024-08-30T19:05:40.473 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:05:40.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757251 2024-08-30T19:05:40.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-08-30T19:05:40.475 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 163208757251 2024-08-30T19:05:40.475 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:05:40.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757249 -lt 163208757251 2024-08-30T19:05:40.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:05:41.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:05:41.874 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:05:42.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757251 -lt 163208757251 2024-08-30T19:05:42.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:05:42.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:42.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:42.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:05:42.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:42.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:43.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:43.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:43.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:43.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:43.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:43.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:05:43.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:05:43.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:05:43.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:43.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:05:43.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:43.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:43.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:43.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:43.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:43.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:43.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:43.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:44.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:44.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:44.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:44.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:44.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:44.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:44.898 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:05:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:05:45.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:45.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:45.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:46.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:46.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:46.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:46.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:46.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:46.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:47.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:47.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:47.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:05:47.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:47.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:05:47.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:47.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:47.574 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:47.574 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:47.574 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:47.574 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:47.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:47.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:48.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:48.003 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:48.003 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:48.003 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:48.571 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:48.571 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:49.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:49.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:49.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:05:49.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:49.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:05:49.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:49.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:49.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:50.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:50.334 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:50.334 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:50.334 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:50.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:50.874 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:50.874 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:51.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:51.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:51.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:05:51.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:51.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:05:53.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:53.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:53.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:53.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:53.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:53.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:53.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:53.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:53.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:53.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:53.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:53.459 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:54.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:54.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:54.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:54.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:54.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:54.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:05:54.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:54.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:05:57.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:05:57.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:05:57.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:05:57.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:05:57.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:05:57.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:05:57.781 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:05:57.781 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:05:58.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:05:58.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:05:58.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:05:58.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:05:58.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:05:58.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:05:58.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:05:58.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:05:59.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:05:59.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:05:59.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:05:59.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:05:59.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:06:05.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:06:05.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:06:05.674 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:06:05.675 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:06:05.675 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:06:05.675 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:06:05.676 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:06:05.676 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:06:06.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:06:06.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:06:06.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:06:06.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:06:06.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:06:06.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:06:06.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:06:06.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:06:06.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:06:06.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:06:06.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:06:06.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:06:06.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:06:07.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:06:07.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:06:07.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:06:07.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:06:07.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:06:14.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:06:14.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:06:14.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:06:14.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:06:14.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:06:14.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:06:14.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:06:14.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:06:14.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:06:14.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:06:14.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:06:14.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:06:15.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:06:15.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:06:16.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:06:16.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:06:16.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:06:16.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:06:16.594 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "ec8faf5c-72c2-4c11-b701-d1d6eaa59179", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:06:16.591921+0000", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:06:16.595 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:06:16.596 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:06:16.597 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:06:16.598 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:06:16.599 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:02:51.391702Z", 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:02:51.391702Z", 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:06:16.600 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:06:16.601 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:06:16.602 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.603 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.604 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 43, 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:02:52.258792+0000", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:05:39.118540+0000", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:05:37.344566+0000", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:05:38.716769+0000", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:06:16.605 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:06:16.606 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:04:37.700979+0000", 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-08-30T19:06:16.607 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:06:16.608 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "29", 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T19:06:16.609 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:06:16.610 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:06:16.611 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-08-30T19:06:16.612 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.613 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "a2fdc2c8-75f3-4a0f-b764-fdf422f32a14", 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:06:16.614 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 36, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 41, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.615 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:06:16.616 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:06:16.617 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:06:16.618 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2553705694 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/2553705694", 2024-08-30T19:06:16.619 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/2553705694", 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2553705694", 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2553705694", 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "112bf508-eac0-4def-a4cd-b016ce41f642", 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:06:16.620 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:06:16.621 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.622 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.623 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.624 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2976352712 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.625 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/2976352712", 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/2976352712", 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2976352712", 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2976352712", 2024-08-30T19:06:16.626 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:06:16.627 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:06:16.627 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.627 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.627 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.627 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "d6e2cee9-0807-4805-a52a-83cda77e491b", 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:06:16.628 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 38, 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.629 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.630 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.631 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.632 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:06:16.633 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.634 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:06:16.635 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.636 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4270653665 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/4270653665", 2024-08-30T19:06:16.637 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/4270653665", 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4270653665", 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4270653665", 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.638 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:05:23.991424+0000", 2024-08-30T19:06:16.639 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:03:25.596514+0000", 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:06:16.640 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:05:24.640366+0000", 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:06:16.641 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:04:00.290151+0000", 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.642 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:05:25.205000+0000", 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:06:16.643 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:04:33.904242+0000", 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:06:16.644 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:06:16.645 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:06:16.646 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:06:16.647 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.648 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/2553705694,v1:127.0.0.1:6805/2553705694]", 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:06:16.649 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:06:16.650 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:06:16.651 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:02:58.908297Z", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:06:16.652 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/2553705694,v1:127.0.0.1:6803/2553705694]", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2553705694,v1:127.0.0.1:6809/2553705694]", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2553705694,v1:127.0.0.1:6807/2553705694]", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:06:16.653 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/2976352712,v1:127.0.0.1:6813/2976352712]", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:06:16.654 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.655 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:03:32.843509Z", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/2976352712,v1:127.0.0.1:6811/2976352712]", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:06:16.656 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2976352712,v1:127.0.0.1:6817/2976352712]", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2976352712,v1:127.0.0.1:6815/2976352712]", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:06:16.657 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/4270653665,v1:127.0.0.1:6813/4270653665]", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:06:16.658 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:06:16.659 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:04:06.990470Z", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/4270653665,v1:127.0.0.1:6811/4270653665]", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4270653665,v1:127.0.0.1:6817/4270653665]", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4270653665,v1:127.0.0.1:6815/4270653665]", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:06:16.660 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:06:16.661 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:06:16.662 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 36 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 10 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 38 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.663 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 43, 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:06:16.664 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:06:16.665 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.666 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.667 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.668 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:06:16.669 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.670 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.671 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:06:16.672 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.673 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:06:16.674 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:06:16.675 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.676 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:06:16.677 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:06:16.678 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:06:16.679 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:02:52:234174+0000", 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:06:16.680 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:06:16.681 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:06:16.682 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:06:16.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:06:16.684 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:06:16.685 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:06:16.686 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:06:16.687 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83708, 2024-08-30T19:06:16.688 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2144, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489092, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036830208, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2195456, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 471158, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:06:16.689 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 1, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.690 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:06:16.691 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83708, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2144, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489092, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036830208, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2195456, 2024-08-30T19:06:16.692 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 471158, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 1, 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:06:16.693 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.694 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:06:16.695 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:06:16.696 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:06:16.697 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:06:16.698 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1228800, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 185100, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:06:16.699 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 154618822667 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672985 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.700 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 163208757258 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-08-30T19:06:16.701 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.702 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.705 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:06:16.706 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.706 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:06:16.706 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:06:16.706 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:06:16.706 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:06:16.707 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:05:27.763639+0000", 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:06:16.708 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 91, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 380, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 520, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:06:16.709 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:06:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:report 722731000 2024-08-30T19:06:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:06:16.710 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-08-30T19:06:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-08-30T19:06:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-08-30T19:06:16.711 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-08-30T19:06:16.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:17.008 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:17.008 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 29'100 43:132 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:17.008 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:17.008 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:17.009 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:17.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-08-30T19:06:17.124 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-08-30T19:06:17.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-08-30T19:06:17.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:17.508 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:17.508 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 29'100 43:132 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:17.508 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:17.508 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:17.508 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:17.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:17.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:17.526 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:06:17.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-08-30T19:06:17.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok set_recovery_delay 100000 2024-08-30T19:06:17.637 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:06:17.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-08-30T19:06:17.650 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-08-30T19:06:17.650 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:06:17.650 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:06:17.651 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:06:17.651 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:17.651 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:17.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:06:17.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-08-30T19:06:17.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok set_recovery_delay 100000 2024-08-30T19:06:17.766 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:06:17.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-08-30T19:06:17.779 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-08-30T19:06:17.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:18.174 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:18.174 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:18.174 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:18.174 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:18.175 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:18.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-08-30T19:06:18.855 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-08-30T19:06:18.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-08-30T19:06:18.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:06:18.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:06:18.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:06:18.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:06:18.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:06:18.881 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:06:18.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:06:18.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:06:18.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:06:18.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:06:18.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:06:18.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:06:18.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:06:18.922 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:18.919+0000 7f52efe0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:18.924 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:18.923+0000 7f52efe0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:18.926 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:18.923+0000 7f52efe0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:19.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-30T19:06:19.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-08-30T19:06:19.986 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:19.983+0000 7f52efe0b7c0 -1 Falling back to public interface 2024-08-30T19:06:23.012 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:23.011+0000 7f52efe0b7c0 -1 osd.1 29 log_to_monitors true 2024-08-30T19:06:24.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-08-30T19:06:24.369 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-08-30T19:06:24.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:24.783 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:24.784 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 45:137 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:24.784 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:24.784 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:24.784 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:24.801 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-08-30T19:06:24.801 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:06:24.801 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:06:24.802 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:06:24.802 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:24.802 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:24.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:06:24.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-08-30T19:06:24.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok set_recovery_delay 100000 2024-08-30T19:06:24.917 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:06:24.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-08-30T19:06:25.566 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-08-30T19:06:25.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-08-30T19:06:25.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:06:25.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:06:25.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:06:25.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:06:26.211 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2553705694,v1:127.0.0.1:6803/2553705694] [v2:127.0.0.1:6804/2553705694,v1:127.0.0.1:6805/2553705694] exists,up a2fdc2c8-75f3-4a0f-b764-fdf422f32a14 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:06:26.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:06:26.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:06:26.667 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1110475997,v1:127.0.0.1:6819/1110475997] [v2:127.0.0.1:6820/1110475997,v1:127.0.0.1:6821/1110475997] exists,up 112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:06:26.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:06:26.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:06:26.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:06:26.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:06:26.669 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:06:26.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:06:26.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:06:26.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:06:27.066 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/4270653665,v1:127.0.0.1:6811/4270653665] [v2:127.0.0.1:6812/4270653665,v1:127.0.0.1:6813/4270653665] exists,up d6e2cee9-0807-4805-a52a-83cda77e491b 2024-08-30T19:06:27.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:06:27.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:06:27.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:06:27.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:27.479 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:27.479 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 45:137 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:27.479 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:27.479 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:27.480 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:27.495 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-08-30T19:06:27.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-08-30T19:06:27.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:27.875 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:27.876 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 45:137 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:27.876 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:27.876 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:27.876 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:27.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-08-30T19:06:28.026 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-08-30T19:06:28.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-08-30T19:06:28.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:28.413 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:28.413 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 45:137 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:28.413 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:28.413 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:28.413 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:28.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-08-30T19:06:28.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:06:28.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:06:28.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:06:28.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:06:28.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:06:28.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:06:28.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-08-30T19:06:28.535 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-08-30T19:06:28.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:28.953 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:28.953 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:05:39.437790+0000 43'101 45:137 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:16:46.045271+0000 0 0 2024-08-30T19:06:28.953 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:28.953 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:28.953 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:28.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:06:28.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:06:28.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:06:28.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:06:28.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:06:28.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:06:28.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:28.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:06:28.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:06:28.971 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:06:28.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:06:28.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:06:28.971 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:06:28.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:06:28.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:06:28.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:06:28.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:06:29.016 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:29.015+0000 7ffbde4327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:29.018 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:29.015+0000 7ffbde4327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:29.020 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:29.019+0000 7ffbde4327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:29.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:06:29.446 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:06:29.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:06:29.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:06:29.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:06:29.861 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1110475997,v1:127.0.0.1:6819/1110475997] [v2:127.0.0.1:6820/1110475997,v1:127.0.0.1:6821/1110475997] exists,up 112bf508-eac0-4def-a4cd-b016ce41f642 2024-08-30T19:06:29.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:06:29.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:06:29.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:06:29.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-08-30T19:06:30.266 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:30.263+0000 7ffbde4327c0 -1 Falling back to public interface 2024-08-30T19:06:33.139 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:33.135+0000 7ffbde4327c0 -1 osd.1 49 log_to_monitors true 2024-08-30T19:06:49.863 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-08-30T19:06:49.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-08-30T19:06:49.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:50.271 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:50.271 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-08-30T19:06:34.623035+0000 49'102 53:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:07:56.136342+0000 0 0 2024-08-30T19:06:50.271 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:50.271 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:50.272 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:50.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-30T19:06:50.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-30T19:06:50.462 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:06:50.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-30T19:06:50.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-30T19:06:50.623 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:06:50.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-30T19:06:50.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-30T19:06:50.775 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:06:50.788 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-08-30T19:06:50.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-08-30T19:06:50.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-08-30T19:06:51.156 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:06:51.156 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-08-30T19:06:34.623035+0000 49'102 53:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:04:37.751263+0000 0'0 2024-08-30T19:04:37.751263+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:07:56.136342+0000 0 0 2024-08-30T19:06:51.156 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:06:51.156 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:06:51.156 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:06:51.172 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-08-30T19:06:51.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-08-30T19:06:51.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:06:51.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-08-30T19:06:51.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:06:51.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-08-30T19:06:51.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:06:51.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-08-30T19:06:51.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:06:51.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-08-30T19:06:51.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-08-30T19:06:51.318 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-08-30T19:06:51.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:06:51.324 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4616.0:1 2024-08-30T19:05:12.144097+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:06:51.324 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4616.0:1 2024-08-30T19:05:12.144097+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4616.0:1 2024-08-30T19:05:12.144097+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:06:26.279+0000 7f52d3d95640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-08-30T19:06:51.325 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-08-30T19:06:51.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:06:51.330 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-08-30T19:06:51.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-08-30T19:06:51.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-08-30T19:06:51.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-30T19:06:51.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-30T19:06:52.307 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-08-30T19:06:52.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-08-30T19:06:52.325 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:06:52.325 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:06:52.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:06:52.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:06:52.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:06:52.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:06:52.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:06:52.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:06:52.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:06:52.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:06:52.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:06:52.642 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:06:52.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:06:52.643 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:06:52.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:06:52.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:06:52.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:06:52.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:06:52.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:06:52.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:06:52.647 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:06:52.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:06:52.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:06:52.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:06:52.668 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:06:52.669 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:52.669 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:52.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:06:52.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:06:52.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:06:52.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:06:52.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:06:52.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:06:52.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:06:52.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:06:52.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:06:52.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:06:52.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:06:52.675 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:06:52.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:06:52.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:06:52.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:06:52.676 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:06:52.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:06:52.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:06:52.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:06:52.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:06:52.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-30T19:06:52.681 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:06:52.681 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:52.681 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:52.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27261 2024-08-30T19:06:52.685 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:06:52.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-30T19:06:52.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:06:52.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-30T19:06:52.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:06:52.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:06:52.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:06:52.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-30T19:06:52.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-30T19:06:53.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:06:53.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:06:53.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:06:53.056 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:06:53.056 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:53.056 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:53.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:53.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:06:53.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:06:53.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:06:53.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:06:53.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:06:53.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:06:53.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:06:53.277 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:06:53.277 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:06:53.277 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:06:53.277 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:06:53.278 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:53.278 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:53.278 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:06:53.278 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:06:53.278 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get fsid 2024-08-30T19:06:53.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:06:53.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:06:53.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:06:53.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:06:53.394 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:06:53.394 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:06:53.394 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:06:53.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get mon_host 2024-08-30T19:06:53.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-08-30T19:06:53.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-30T19:06:53.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:06:53.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:06:53.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:06:53.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-30T19:06:53.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:06:54.403 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:06:54.403 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:06:54.403 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:06:54.403 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:06:54.404 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:54.404 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:54.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:54.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:06:54.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:06:54.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-30T19:06:54.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:06:54.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:06:54.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:06:54.446 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:06:54.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:06:54.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c984e51a-1348-4ed0-988c-566bb29481d0' 2024-08-30T19:06:54.447 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:06:54.447 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:06:54.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOGNJmmFLjGxAAt6spHgonPn30C1ROxu0zqg== 2024-08-30T19:06:54.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOGNJmmFLjGxAAt6spHgonPn30C1ROxu0zqg=="}' 2024-08-30T19:06:54.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c984e51a-1348-4ed0-988c-566bb29481d0 -i td/divergent-priors/0/new.json 2024-08-30T19:06:54.761 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:06:54.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-30T19:06:54.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOGNJmmFLjGxAAt6spHgonPn30C1ROxu0zqg== --osd-uuid c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:06:54.820 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:54.819+0000 7ff86251a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:54.830 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:54.827+0000 7ff86251a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:54.831 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:54.831+0000 7ff86251a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:06:54.832 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:54.831+0000 7ff86251a7c0 -1 bdev(0x562e28e9aa80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:06:54.832 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:06:54.831+0000 7ff86251a7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-30T19:07:00.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-30T19:07:00.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:07:00.818 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:07:00.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:07:00.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:07:01.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:07:01.389 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:07:01.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:07:01.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:07:01.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:07:01.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:07:01.428 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:01.427+0000 7ffb642557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:01.447 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:01.443+0000 7ffb642557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:01.448 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:01.447+0000 7ffb642557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:07:01.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:01.801 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:07:01.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:07:01.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:01.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:02.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:02.698 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:02.695+0000 7ffb642557c0 -1 Falling back to public interface 2024-08-30T19:07:03.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:03.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:03.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:07:03.217 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:07:03.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:03.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:03.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:04.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:04.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:04.637 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:07:04.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:07:04.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:04.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:05.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:05.761 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:05.759+0000 7ffb642557c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:07:06.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:06.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:06.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:07:06.105 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:07:06.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:06.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:06.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:07.524 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:07:07.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:07:07.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:07.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:07.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:08.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:08.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:08.983 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:07:08.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:07:08.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:08.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:09.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:10.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:10.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:10.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:07:10.500 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:07:10.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:10.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:10.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:11.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:11.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:11.960 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:07:11.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:07:11.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:11.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:12.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:13.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:13.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:13.532 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:07:13.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:07:13.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:13.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:13.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:14.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:14.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:14.974 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:07:14.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:07:14.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:14.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:15.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:16.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:16.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:16.532 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:07:16.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:07:16.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:16.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:16.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:17.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:17.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:17.967 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:07:17.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:07:17.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:17.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:18.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:19.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:19.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:19.643 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:07:19.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:07:19.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:19.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:20.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:21.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:21.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:21.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:07:21.327 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:07:21.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:21.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:22.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:23.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:23.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:23.282 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:07:23.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:07:23.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:23.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:24.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:25.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:25.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:25.030 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:07:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:07:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:26.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:27.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:27.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:27.100 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:07:27.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:07:27.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:27.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:27.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:28.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:28.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:28.705 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:07:28.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:07:28.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:28.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:07:29.202 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2428149084,v1:127.0.0.1:6803/2428149084] [v2:127.0.0.1:6804/2428149084,v1:127.0.0.1:6805/2428149084] exists,up c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:07:29.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:07:29.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:07:29.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:07:29.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-30T19:07:29.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:07:29.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:07:29.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:07:29.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:07:29.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:07:29.210 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:07:29.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0b137e2a-5f55-4bf4-885d-48c5c7c60eca' 2024-08-30T19:07:29.210 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:07:29.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxGNJm7XUIDhAAEfGq2f8TrsvzejzRdgzJCQ== 2024-08-30T19:07:29.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxGNJm7XUIDhAAEfGq2f8TrsvzejzRdgzJCQ=="}' 2024-08-30T19:07:29.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b137e2a-5f55-4bf4-885d-48c5c7c60eca -i td/divergent-priors/1/new.json 2024-08-30T19:07:29.752 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:07:29.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-30T19:07:29.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxGNJm7XUIDhAAEfGq2f8TrsvzejzRdgzJCQ== --osd-uuid 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:07:29.812 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:29.811+0000 7f9131d7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:29.815 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:29.815+0000 7f9131d7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:29.816 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:29.815+0000 7f9131d7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:29.816 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:29.815+0000 7f9131d7b7c0 -1 bdev(0x563adfcbea80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:07:29.817 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:29.815+0000 7f9131d7b7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-30T19:07:35.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-30T19:07:35.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:07:35.432 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:07:35.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:07:35.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:07:36.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:07:36.018 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:07:36.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:07:36.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:07:36.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:07:36.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:07:36.059 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:36.058+0000 7fca244b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:36.061 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:36.058+0000 7fca244b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:36.062 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:36.058+0000 7fca244b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:07:36.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:07:36.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:07:36.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:07:36.413 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:36.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:36.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:37.125 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:37.122+0000 7fca244b17c0 -1 Falling back to public interface 2024-08-30T19:07:37.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:37.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:37.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:07:37.892 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:07:37.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:37.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:38.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:07:39.360 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:07:39.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:39.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:39.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:40.002 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:07:39.998+0000 7fca244b17c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:07:40.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:40.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:40.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:07:40.858 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:07:40.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:40.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:41.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:42.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:42.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:42.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:07:42.312 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:07:42.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:42.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:42.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:43.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:43.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:43.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:07:43.834 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:07:43.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:43.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:44.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:45.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:45.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:45.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:07:45.259 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:07:45.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:45.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:45.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:46.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:46.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:46.782 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:07:46.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:07:46.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:46.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:47.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:48.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:48.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:48.237 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:07:48.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:07:48.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:48.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:48.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:07:49.798 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:07:49.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:49.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:50.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:51.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:51.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:51.208 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:07:51.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:07:51.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:51.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:51.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:52.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:52.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:52.994 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:07:52.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:07:52.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:52.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:53.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:54.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:54.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:54.840 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:07:54.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:07:54.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:54.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:55.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:56.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:56.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:56.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:07:56.722 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:07:56.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:56.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:57.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:07:58.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:07:58.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:07:58.523 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:07:58.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:07:58.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:07:58.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:07:59.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:00.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:00.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:00.228 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:08:00.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:08:00.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:00.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:08:00.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:01.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:01.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:01.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:08:01.983 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:08:01.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:01.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:08:02.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:03.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:03.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:03.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:08:03.494 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:08:03.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:03.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:08:03.986 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/241733523,v1:127.0.0.1:6811/241733523] [v2:127.0.0.1:6812/241733523,v1:127.0.0.1:6813/241733523] exists,up 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:08:03.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:08:03.990 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:08:03.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:08:03.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:08:03.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:08:03.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:08:03.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 21bc4db4-4fb0-41e5-9548-c0193afa3f30' 2024-08-30T19:08:03.994 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:08:03.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:08:04.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUGNJmo4ZGARAAWtPQH3yBwC5p1j/gQgSJTQ== 2024-08-30T19:08:04.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUGNJmo4ZGARAAWtPQH3yBwC5p1j/gQgSJTQ=="}' 2024-08-30T19:08:04.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21bc4db4-4fb0-41e5-9548-c0193afa3f30 -i td/divergent-priors/2/new.json 2024-08-30T19:08:04.506 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:08:04.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-30T19:08:04.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUGNJmo4ZGARAAWtPQH3yBwC5p1j/gQgSJTQ== --osd-uuid 21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:08:04.567 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:04.566+0000 7ff6b045d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:04.569 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:04.566+0000 7ff6b045d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:04.570 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:04.570+0000 7ff6b045d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:04.570 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:04.570+0000 7ff6b045d7c0 -1 bdev(0x55ba3272aa80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:08:04.571 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:04.570+0000 7ff6b045d7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-30T19:08:10.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-30T19:08:10.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:08:10.777 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:08:10.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:08:10.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:08:11.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:08:11.341 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:08:11.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:08:11.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:08:11.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:08:11.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:08:11.383 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:11.382+0000 7fd38cb687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:11.407 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:11.406+0000 7fd38cb687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:11.408 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:11.406+0000 7fd38cb687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:08:11.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:08:11.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:08:11.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:08:11.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:08:11.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:08:11.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:11.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:08:11.762 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:08:11.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:11.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:12.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:12.665 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:12.662+0000 7fd38cb687c0 -1 Falling back to public interface 2024-08-30T19:08:13.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:13.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:13.212 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:08:13.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:08:13.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:13.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:13.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:14.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:14.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:14.643 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:08:14.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:08:14.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:14.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:15.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:15.486 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:08:15.482+0000 7fd38cb687c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:08:16.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:16.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:16.061 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:08:16.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:08:16.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:16.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:16.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:17.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:17.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:17.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:08:17.588 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:08:17.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:17.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:18.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:19.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:19.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:19.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:08:19.097 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:08:19.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:19.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:19.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:20.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:20.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:20.542 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:08:20.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:08:20.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:20.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:21.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:22.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:22.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:22.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:08:22.035 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:08:22.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:22.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:22.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:23.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:23.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:23.508 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:08:23.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:08:23.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:23.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:23.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:24.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:24.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:24.937 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:08:24.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:08:24.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:24.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:25.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:26.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:26.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:26.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:08:26.495 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:08:26.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:26.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:26.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:27.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:27.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:27.938 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:08:27.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:08:27.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:27.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:28.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:29.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:29.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:29.562 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:08:29.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:08:29.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:29.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:30.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:31.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:31.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:31.353 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:08:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:08:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:32.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:33.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:33.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:33.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:08:33.110 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:08:33.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:33.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:33.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:34.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:34.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:34.834 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:08:34.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:08:34.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:34.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:35.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:36.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:36.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:36.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:08:36.448 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:08:36.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:36.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:37.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:38.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:38.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:38.014 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:08:38.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:08:38.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:38.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:38.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:08:39.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:08:39.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:08:39.475 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:08:39.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:08:39.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:08:39.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:08:39.985 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3945697421,v1:127.0.0.1:6819/3945697421] [v2:127.0.0.1:6820/3945697421,v1:127.0.0.1:6821/3945697421] exists,up 21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:08:39.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:08:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:08:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:08:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-08-30T19:08:40.752 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-08-30T19:08:40.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-08-30T19:08:41.680 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-08-30T19:08:41.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-08-30T19:08:42.434 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-08-30T19:08:42.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-08-30T19:08:42.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-30T19:08:43.189 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-08-30T19:08:43.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:08:44.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-08-30T19:08:44.887 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-08-30T19:08:44.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-08-30T19:08:45.750 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-08-30T19:08:45.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-08-30T19:08:45.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:08:45.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:08:46.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:08:46.222 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:08:46.222 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:08:46.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:08:46.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:46.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:08:46.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-08-30T19:08:46.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-08-30T19:08:46.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-08-30T19:08:46.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:46.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:08:46.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-08-30T19:08:46.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-08-30T19:08:46.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-08-30T19:08:46.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:46.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509448' 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-08-30T19:08:46.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:46.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:08:46.711 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-08-30T19:08:46.711 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:46.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-08-30T19:08:46.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-30T19:08:46.712 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836503 2024-08-30T19:08:46.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:08:47.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-08-30T19:08:47.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:47.128 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:47.128 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-08-30T19:08:47.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:08:47.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-08-30T19:08:47.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:47.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-08-30T19:08:47.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-08-30T19:08:47.131 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672974 2024-08-30T19:08:47.132 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:08:47.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-08-30T19:08:47.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:47.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-08-30T19:08:47.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:08:47.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-08-30T19:08:47.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:47.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-08-30T19:08:47.558 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509448 2024-08-30T19:08:47.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-08-30T19:08:47.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:08:48.028 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:08:48.029 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:08:48.029 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:08:48.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:08:48.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:08:48.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:08:48.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:08:48.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:08:48.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:08:48.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:08:48.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:08:48.137 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:08:48.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:08:48.604 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:08:48.604 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:08:48.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:08:48.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:48.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:08:48.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-08-30T19:08:48.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-08-30T19:08:48.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-08-30T19:08:48.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:48.777 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:08:48.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-30T19:08:48.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-30T19:08:48.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-08-30T19:08:48.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:08:48.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:08:49.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-08-30T19:08:49.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-08-30T19:08:49.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509451' 2024-08-30T19:08:49.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:49.092 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-08-30T19:08:49.092 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:49.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:08:49.094 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-08-30T19:08:49.094 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:49.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-08-30T19:08:49.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-08-30T19:08:49.095 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836506 2024-08-30T19:08:49.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:08:49.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-08-30T19:08:49.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:49.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-30T19:08:49.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:49.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:08:49.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-30T19:08:49.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:49.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-30T19:08:49.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-30T19:08:49.520 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672977 2024-08-30T19:08:49.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:08:49.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-30T19:08:49.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:08:49.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-08-30T19:08:49.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:08:49.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:08:49.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-08-30T19:08:49.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:08:49.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-08-30T19:08:49.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-08-30T19:08:49.935 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509451 2024-08-30T19:08:49.935 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:08:50.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509451 2024-08-30T19:08:50.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:08:51.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:08:51.418 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:08:51.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-08-30T19:08:51.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:08:51.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:08:51.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:08:52.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:08:52.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:08:52.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:08:52.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:08:52.838 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:08:52.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:08:52.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:08:53.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:08:53.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:08:53.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:08:53.371 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-08-30T19:08:53.371 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-08-30T19:08:53.805 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:08:53.821 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-08-30T19:08:53.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-08-30T19:08:53.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-08-30T19:08:53.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:08:54.205 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:08:54.205 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:08:44.496654+0000 0'0 28:20 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:51:50.231351+0000 0 0 2024-08-30T19:08:54.205 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:08:54.205 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:08:54.205 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-08-30T19:08:54.220 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-08-30T19:08:54.221 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-08-30T19:08:54.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:54.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-08-30T19:08:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-08-30T19:08:54.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:54.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-08-30T19:08:54.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:54.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-08-30T19:08:54.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:54.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-08-30T19:08:55.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-08-30T19:08:55.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-08-30T19:08:55.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-08-30T19:08:55.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-08-30T19:08:55.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-08-30T19:08:55.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:55.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-08-30T19:08:56.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-08-30T19:08:56.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-08-30T19:08:56.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-08-30T19:08:56.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-08-30T19:08:56.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-08-30T19:08:56.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-08-30T19:08:56.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:56.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-08-30T19:08:57.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-08-30T19:08:57.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-08-30T19:08:57.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-08-30T19:08:57.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-08-30T19:08:57.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-08-30T19:08:57.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:57.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-08-30T19:08:58.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-08-30T19:08:58.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-08-30T19:08:58.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-08-30T19:08:58.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-08-30T19:08:58.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-08-30T19:08:58.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:58.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-08-30T19:08:59.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-08-30T19:08:59.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-08-30T19:08:59.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-08-30T19:08:59.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-08-30T19:08:59.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-08-30T19:08:59.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-08-30T19:08:59.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:08:59.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-08-30T19:09:00.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-08-30T19:09:00.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-08-30T19:09:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-08-30T19:09:00.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-08-30T19:09:00.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-08-30T19:09:00.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:00.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-08-30T19:09:01.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-08-30T19:09:01.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-08-30T19:09:01.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-08-30T19:09:01.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-08-30T19:09:01.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-08-30T19:09:01.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:01.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-08-30T19:09:02.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-08-30T19:09:02.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-08-30T19:09:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-08-30T19:09:02.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-08-30T19:09:02.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-08-30T19:09:02.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-08-30T19:09:02.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:02.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-08-30T19:09:03.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-08-30T19:09:03.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-08-30T19:09:03.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-08-30T19:09:03.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-08-30T19:09:03.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-08-30T19:09:03.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-08-30T19:09:04.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-08-30T19:09:04.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-08-30T19:09:04.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-08-30T19:09:04.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-08-30T19:09:04.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-08-30T19:09:04.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-08-30T19:09:04.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:04.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-08-30T19:09:05.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:05.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-08-30T19:09:05.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:05.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-08-30T19:09:05.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:05.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-08-30T19:09:05.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:05.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-08-30T19:09:05.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:05.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-08-30T19:09:06.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-08-30T19:09:06.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-08-30T19:09:06.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-08-30T19:09:06.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-08-30T19:09:06.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-08-30T19:09:06.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-08-30T19:09:06.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:06.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-08-30T19:09:07.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-08-30T19:09:07.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-08-30T19:09:07.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-08-30T19:09:07.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-08-30T19:09:07.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-08-30T19:09:07.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:07.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-08-30T19:09:08.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-08-30T19:09:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-08-30T19:09:08.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-08-30T19:09:08.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-08-30T19:09:08.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-08-30T19:09:08.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:08.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-08-30T19:09:09.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-08-30T19:09:09.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-08-30T19:09:09.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-08-30T19:09:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-08-30T19:09:09.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-08-30T19:09:09.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-08-30T19:09:09.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-30T19:09:09.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:09:10.114 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:09:10.115 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:09:10.115 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:09:10.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:09:10.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:09:10.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:09:10.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:09:10.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:09:10.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:09:10.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:09:10.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:09:10.192 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:10.624 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:09:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-08-30T19:09:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-08-30T19:09:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-08-30T19:09:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:10.789 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:09:10.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-08-30T19:09:10.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-08-30T19:09:10.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-08-30T19:09:10.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:10.948 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:09:11.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509457' 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:09:11.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:09:11.104 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-08-30T19:09:11.104 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:09:11.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-08-30T19:09:11.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-08-30T19:09:11.106 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836512 2024-08-30T19:09:11.106 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:09:11.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-08-30T19:09:11.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:09:11.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-08-30T19:09:11.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:09:11.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:09:11.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-08-30T19:09:11.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:09:11.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-08-30T19:09:11.562 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672983 2024-08-30T19:09:11.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-30T19:09:11.563 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:09:12.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-08-30T19:09:12.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:09:12.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-08-30T19:09:12.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:09:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:09:12.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-08-30T19:09:12.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:09:12.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-08-30T19:09:12.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-30T19:09:12.047 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509457 2024-08-30T19:09:12.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:09:12.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509457 2024-08-30T19:09:12.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:09:13.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:09:13.521 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:09:13.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-30T19:09:13.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:09:13.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:13.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:14.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:09:14.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:14.518 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:14.518 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:14.518 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:14.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:14.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:14.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:14.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:09:14.927 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:14.928 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:14.928 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-30T19:09:15.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-30T19:09:15.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-08-30T19:09:15.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-08-30T19:09:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-08-30T19:09:15.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:09:16.324 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:09:16.325 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-30T19:08:44.496654+0000 28'100 28:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:51:50.231351+0000 0 0 2024-08-30T19:09:16.325 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:09:16.325 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:09:16.325 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:09:16.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:16.342 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-08-30T19:09:16.342 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:09:16.342 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:09:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-30T19:09:16.438 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:09:16.438 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:09:16.438 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:09:16.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:09:16.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:09:16.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:09:16.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-30T19:09:16.554 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:09:16.554 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:09:16.555 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:09:16.566 INFO:tasks.workunit.client.0.smithi079.stdout:create new divergent objects 2024-08-30T19:09:16.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-08-30T19:09:16.567 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-08-30T19:09:16.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:09:16.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:09:16.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-08-30T19:09:16.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-08-30T19:09:16.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-08-30T19:09:16.570 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent objects 2024-08-30T19:09:16.570 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-08-30T19:09:16.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-30T19:09:16.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-30T19:09:16.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-08-30T19:09:16.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-08-30T19:09:16.572 INFO:tasks.workunit.client.0.smithi079.stdout:remove divergent objects 2024-08-30T19:09:16.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-08-30T19:09:16.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-08-30T19:09:16.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-30T19:09:16.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-08-30T19:09:16.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-08-30T19:09:16.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-30T19:09:16.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-08-30T19:09:16.577 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-08-30T19:09:16.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-08-30T19:09:16.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-08-30T19:09:16.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-08-30T19:09:26.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-08-30T19:09:26.584 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-08-30T19:09:26.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-08-30T19:09:26.585 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-08-30T19:09:26.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:09:27.000 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:09:27.000 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-08-30T19:08:44.496654+0000 28'106 28:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:51:50.231351+0000 0 0 2024-08-30T19:09:27.000 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:09:27.000 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:09:27.000 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:09:27.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-08-30T19:09:27.017 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:09:27.017 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:09:27.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:09:27.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:09:27.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:09:27.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:09:27.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-30T19:09:27.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-08-30T19:09:27.663 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:09:27.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-30T19:09:27.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-08-30T19:09:28.223 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:09:28.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-30T19:09:28.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-08-30T19:09:28.791 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:09:28.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:28.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-08-30T19:09:29.431 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-08-30T19:09:29.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:29.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-08-30T19:09:29.994 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-08-30T19:09:30.013 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-08-30T19:09:30.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-08-30T19:09:30.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:09:30.406 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:09:30.406 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-08-30T19:08:44.496654+0000 28'106 28:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:51:50.231351+0000 0 0 2024-08-30T19:09:30.406 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:09:30.406 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:09:30.407 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:09:30.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:30.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:09:30.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:09:30.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:09:30.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:09:30.426 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:09:30.426 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:09:30.426 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:09:30.427 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:09:30.427 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:09:30.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:09:30.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:09:30.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:09:30.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:09:30.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:09:30.429 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:09:30.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:09:30.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:09:30.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:09:30.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:09:30.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:09:30.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:09:30.467 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:30.465+0000 7fd57bcbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:30.469 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:30.469+0000 7fd57bcbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:30.470 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:30.469+0000 7fd57bcbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:30.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:09:30.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:09:30.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:09:30.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:09:30.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:09:30.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:30.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:09:30.940 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:09:30.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:09:30.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:31.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:31.503 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:31.501+0000 7fd57bcbd7c0 -1 Falling back to public interface 2024-08-30T19:09:32.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:32.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:32.404 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:09:32.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:09:32.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:32.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:09:32.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:33.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:33.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:33.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:09:33.861 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:09:33.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:33.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:09:34.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:34.659 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:34.657+0000 7fd57bcbd7c0 -1 osd.0 28 log_to_monitors true 2024-08-30T19:09:35.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:35.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:35.282 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:09:35.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:09:35.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:35.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:09:35.730 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/353272593,v1:127.0.0.1:6803/353272593] [v2:127.0.0.1:6804/353272593,v1:127.0.0.1:6805/353272593] exists,up c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:09:35.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:09:35.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:09:35.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:09:35.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:09:35.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:09:35.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:09:35.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:09:35.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:09:35.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:09:35.737 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:09:35.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:09:35.775 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:35.773+0000 7f37135207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:35.777 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:35.777+0000 7f37135207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:35.778 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:35.777+0000 7f37135207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:09:36.237 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:09:36.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:36.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:09:36.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:36.780 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:36.777+0000 7f37135207c0 -1 Falling back to public interface 2024-08-30T19:09:37.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:37.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:37.657 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:09:37.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:09:37.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:37.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:09:38.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:39.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:39.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:39.068 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:09:39.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:09:39.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:39.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:09:39.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:09:39.997 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:09:39.993+0000 7f37135207c0 -1 osd.2 28 log_to_monitors true 2024-08-30T19:09:40.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:09:40.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:09:40.481 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:09:40.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:09:40.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:09:40.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:09:40.958 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/2578225128,v1:127.0.0.1:6811/2578225128] [v2:127.0.0.1:6812/2578225128,v1:127.0.0.1:6813/2578225128] exists,up 21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:09:40.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:09:40.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:09:40.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:09:40.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:40.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-08-30T19:09:41.487 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-08-30T19:09:41.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:09:41.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-08-30T19:09:42.447 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-08-30T19:09:42.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:09:42.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-08-30T19:09:42.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:09:42.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:09:42.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:09:42.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:09:42.474 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:09:42.474 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:09:42.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:09:42.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:09:42.475 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:09:42.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:09:42.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:09:42.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:09:42.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:09:42.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:09:42.549 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:09:42.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:09:42.993 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:09:42.993 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:09:42.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:09:42.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:42.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:09:43.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-08-30T19:09:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-08-30T19:09:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364' 2024-08-30T19:09:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:43.184 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:09:43.310 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T19:09:43.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:09:43.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:09:43.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:09:43.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:09:43.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:09:43.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789955 2024-08-30T19:09:43.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789955 2024-08-30T19:09:43.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2024-08-30T19:09:43.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:09:43.477 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855364 2024-08-30T19:09:43.477 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:09:43.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:09:43.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855364 2024-08-30T19:09:43.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:09:43.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-08-30T19:09:43.480 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 150323855364 2024-08-30T19:09:43.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-08-30T19:09:43.480 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:09:43.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855364 2024-08-30T19:09:43.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:09:44.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:09:44.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:09:45.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855364 2024-08-30T19:09:45.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:09:45.401 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789955 2024-08-30T19:09:45.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:09:45.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:09:45.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789955 2024-08-30T19:09:45.405 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:09:45.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789955 2024-08-30T19:09:45.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-08-30T19:09:45.406 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 158913789955 2024-08-30T19:09:45.406 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:09:45.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789955 -lt 158913789955 2024-08-30T19:09:45.831 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:09:45.831 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:45.832 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:46.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:09:46.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:46.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:46.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:46.763 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:46.764 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:46.764 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:47.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:09:47.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:09:47.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:47.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:47.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:47.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:47.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:47.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:47.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:47.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:47.846 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:47.846 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:47.846 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:48.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:48.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:09:48.383 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:09:48.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:09:48.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:09:48.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:09:48.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:48.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:09:49.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:09:49.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:49.140 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:49.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:49.570 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:49.570 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:49.570 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:50.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:50.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:09:50.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:09:50.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:09:50.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:09:50.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:09:50.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:09:50.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:09:50.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:09:50.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:09:50.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:09:50.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:09:50.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:50.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:09:51.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:09:51.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:51.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:51.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:51.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:51.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:51.064 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:51.064 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:51.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:51.468 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:51.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:51.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:52.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:09:52.018 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:09:52.019 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:09:52.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:09:52.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:09:52.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:09:52.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:52.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:09:53.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:09:53.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:53.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:53.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:53.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:53.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:53.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:53.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:53.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:53.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:53.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:53.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:09:54.305 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:09:54.306 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:09:54.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:09:54.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:09:54.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:09:54.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:54.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:09:56.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:09:56.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:09:56.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:09:56.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:09:56.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:09:56.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:09:56.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:09:57.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:09:57.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:09:57.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:09:57.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:09:57.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:09:57.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:09:57.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:10:01.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:01.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:01.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:01.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:01.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:01.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:01.162 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:01.162 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:01.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:01.577 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:01.578 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:01.578 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:02.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:02.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:02.138 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:02.138 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:02.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:02.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:02.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:10:02.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:02.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:10:09.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:09.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:09.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:09.081 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:09.081 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:09.081 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:09.081 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:09.081 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:09.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:09.501 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:09.502 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:09.502 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:10.053 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:10.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:10.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:10.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:10:10.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:10.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:10:17.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:17.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:17.898 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:17.898 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:17.899 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:17.899 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:17.899 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:17.899 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:18.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:18.327 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:18.328 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:18.328 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:18.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:18.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:18.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:19.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:19.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:19.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:10:19.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:10:19.953 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:10:19.953 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "08b04f38-9269-4cd8-a509-5be49ecce329", 2024-08-30T19:10:19.953 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:10:19.953 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:10:19.954596+0000", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:19.954 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:19.955 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.956 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.957 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:06:52.710612Z", 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:06:52.710612Z", 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:10:19.958 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:10:19.959 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:19.960 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:10:19.961 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 41, 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:06:53.575772+0000", 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:09:42.291257+0000", 2024-08-30T19:10:19.962 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:09:40.304746+0000", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:09:41.999628+0000", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-08-30T19:10:19.963 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:10:19.964 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:08:42.830503+0000", 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:10:19.965 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:10:19.966 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "28", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-08-30T19:10:19.967 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:10:19.968 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:10:19.969 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-08-30T19:10:19.970 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.971 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "c984e51a-1348-4ed0-988c-566bb29481d0", 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 28, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 35, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 40, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 29, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.972 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.973 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.974 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.975 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:19.976 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/353272593", 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/353272593", 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/353272593", 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/353272593", 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.977 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "0b137e2a-5f55-4bf4-885d-48c5c7c60eca", 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.978 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.979 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.980 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.981 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/241733523", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/241733523", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/241733523", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/241733523", 2024-08-30T19:10:19.982 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "21bc4db4-4fb0-41e5-9548-c0193afa3f30", 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 28, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 37, 2024-08-30T19:10:19.983 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.984 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.985 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:10:19.986 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:19.987 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/2578225128", 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/2578225128", 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2578225128", 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2578225128", 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.988 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:27.487476+0000", 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:07:27.405690+0000", 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:28.046312+0000", 2024-08-30T19:10:19.989 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:08:02.048739+0000", 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:28.606038+0000", 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:19.990 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:08:38.061768+0000", 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:10:19.991 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:10:19.992 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/353272593,v1:127.0.0.1:6805/353272593]", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:19.993 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:19.994 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:07:00.144741Z", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/353272593,v1:127.0.0.1:6803/353272593]", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:19.995 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/353272593,v1:127.0.0.1:6809/353272593]", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/353272593,v1:127.0.0.1:6807/353272593]", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:19.996 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/241733523,v1:127.0.0.1:6813/241733523]", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:19.997 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:07:34.766483Z", 2024-08-30T19:10:19.998 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/241733523,v1:127.0.0.1:6811/241733523]", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/241733523,v1:127.0.0.1:6817/241733523]", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/241733523,v1:127.0.0.1:6815/241733523]", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:19.999 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:20.000 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/2578225128,v1:127.0.0.1:6813/2578225128]", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:10:20.001 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:08:10.121261Z", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:20.002 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/2578225128,v1:127.0.0.1:6811/2578225128]", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2578225128,v1:127.0.0.1:6817/2578225128]", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2578225128,v1:127.0.0.1:6815/2578225128]", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:20.003 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.004 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 35 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 10 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 37 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.005 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 41, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:20.006 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.007 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:10:20.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:10:20.010 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:10:20.011 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.012 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:20.013 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:10:20.014 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.015 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:10:20.016 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:10:20.017 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:10:20.018 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:10:20.019 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:06:53:543315+0000", 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:10:20.020 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stderr:report 3237633011 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:10:20.021 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:10:20.022 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.023 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:10:20.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:10:20.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:10:20.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:20.027 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:10:20.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2096, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2146304, 2024-08-30T19:10:20.029 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 456078, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:10:20.030 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:10:20.031 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2096, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2146304, 2024-08-30T19:10:20.032 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 456078, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:10:20.033 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:10:20.034 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:10:20.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:10:20.036 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:10:20.037 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 184566, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:10:20.038 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 150323855371 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672986 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:20.039 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 158913789962 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:20.040 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.041 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:09:30.991647+0000", 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 92, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:10:20.042 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 386, 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 529, 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:10:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:10:20.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-08-30T19:10:20.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-08-30T19:10:20.044 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-08-30T19:10:20.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-08-30T19:10:20.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:10:20.345 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:10:20.345 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 28'100 41:129 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:10:20.345 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:10:20.345 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:10:20.345 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:10:20.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:10:20.465 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:10:20.466 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:10:20.466 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:10:20.466 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:10:20.466 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:10:20.467 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:10:20.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:10:20.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:10:20.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:10:20.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:10:20.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:10:20.535 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:10:20.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:10:20.956 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:10:20.956 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:10:20.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:10:20.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:10:20.957 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:10:21.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855374 2024-08-30T19:10:21.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855374 2024-08-30T19:10:21.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855374' 2024-08-30T19:10:21.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:10:21.121 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:10:21.238 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T19:10:21.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:10:21.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:10:21.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:10:21.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:10:21.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:10:21.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789965 2024-08-30T19:10:21.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789965 2024-08-30T19:10:21.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855374 2-158913789965' 2024-08-30T19:10:21.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:10:21.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855374 2024-08-30T19:10:21.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:10:21.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:10:21.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855374 2024-08-30T19:10:21.394 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:10:21.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855374 2024-08-30T19:10:21.395 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 150323855374 2024-08-30T19:10:21.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855374' 2024-08-30T19:10:21.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:10:21.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855372 -lt 150323855374 2024-08-30T19:10:21.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:10:22.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:10:22.806 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:10:23.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855374 -lt 150323855374 2024-08-30T19:10:23.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:10:23.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789965 2024-08-30T19:10:23.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:10:23.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:10:23.242 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789965 2024-08-30T19:10:23.242 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:10:23.243 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 158913789965 2024-08-30T19:10:23.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789965 2024-08-30T19:10:23.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789965' 2024-08-30T19:10:23.244 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:10:23.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789965 -lt 158913789965 2024-08-30T19:10:23.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:10:23.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:23.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:24.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:10:24.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:24.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:24.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:24.608 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:24.609 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:24.609 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:25.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:25.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:10:25.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:10:25.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:10:25.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:25.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:10:25.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:25.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:25.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:25.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:25.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:25.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:25.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:25.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:25.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:25.669 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:25.670 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:25.670 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:26.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:26.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:26.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:26.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:26.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:10:26.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:26.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:10:26.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:26.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:26.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:26.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:26.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:26.932 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:26.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:26.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:27.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:27.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:27.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:27.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:27.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:27.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:27.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:27.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:10:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:10:28.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:28.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:28.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:28.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:28.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:28.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:28.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:28.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:29.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:29.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:29.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:29.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:29.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:29.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:10:30.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:30.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:10:31.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:31.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:31.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:31.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:31.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:31.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:31.151 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:31.151 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:31.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:31.566 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:31.566 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:31.566 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:32.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:32.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:32.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:32.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:32.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:32.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:10:32.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:32.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:10:34.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:34.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:34.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:34.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:34.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:34.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:34.241 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:34.241 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:34.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:34.647 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:34.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:34.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:35.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:35.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:35.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:35.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:35.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:35.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:35.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:35.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:35.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:35.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:35.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:35.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:10:35.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:35.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:10:38.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:38.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:38.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:39.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:39.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:39.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:39.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:39.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:39.999 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:40.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:40.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:40.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:10:40.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:40.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:10:46.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:46.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:46.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:46.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:46.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:46.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:46.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:46.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:47.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:47.332 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:47.332 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:47.332 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:47.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:47.891 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:48.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:48.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:48.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:10:48.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:10:48.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:10:55.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:10:55.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:10:55.737 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:10:55.737 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:10:55.737 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:10:55.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:10:55.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:10:55.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:10:56.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:10:56.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:10:56.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:10:56.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:10:56.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:10:56.692 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:10:57.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:10:57.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:10:57.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:10:57.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "08b04f38-9269-4cd8-a509-5be49ecce329", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:10:57.789474+0000", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:10:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 2 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:10:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:06:52.710612Z", 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:06:52.710612Z", 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:10:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:10:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 41, 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:06:53.575772+0000", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:09:42.291257+0000", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:09:40.304746+0000", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:09:41.999628+0000", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:10:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:10:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:08:42.830503+0000", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:10:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-08-30T19:10:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "28", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T19:10:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:10:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:10:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:10:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "c984e51a-1348-4ed0-988c-566bb29481d0", 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 28, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 35, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 40, 2024-08-30T19:10:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 29, 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:10:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:10:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 353272593 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/353272593", 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/353272593", 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/353272593", 2024-08-30T19:10:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/353272593", 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "0b137e2a-5f55-4bf4-885d-48c5c7c60eca", 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 241733523 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/241733523", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/241733523", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/241733523", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/241733523", 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:10:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "21bc4db4-4fb0-41e5-9548-c0193afa3f30", 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 28, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 37, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-08-30T19:10:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2578225128 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/2578225128", 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/2578225128", 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2578225128", 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2578225128", 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:27.487476+0000", 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:07:27.405690+0000", 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:28.046312+0000", 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:08:02.048739+0000", 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:09:28.606038+0000", 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:10:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:08:38.061768+0000", 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:10:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:10:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/353272593,v1:127.0.0.1:6805/353272593]", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:07:00.144741Z", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/353272593,v1:127.0.0.1:6803/353272593]", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/353272593,v1:127.0.0.1:6809/353272593]", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/353272593,v1:127.0.0.1:6807/353272593]", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/241733523,v1:127.0.0.1:6813/241733523]", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:07:34.766483Z", 2024-08-30T19:10:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/241733523,v1:127.0.0.1:6811/241733523]", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/241733523,v1:127.0.0.1:6817/241733523]", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/241733523,v1:127.0.0.1:6815/241733523]", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:10:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/2578225128,v1:127.0.0.1:6813/2578225128]", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:10:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:08:10.121261Z", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:10:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/2578225128,v1:127.0.0.1:6811/2578225128]", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2578225128,v1:127.0.0.1:6817/2578225128]", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2578225128,v1:127.0.0.1:6815/2578225128]", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:10:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:10:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 35 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 10 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 37 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 41, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:10:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:10:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:10:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:10:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:10:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:10:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:10:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:10:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stderr:report 1166853046 2024-08-30T19:10:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:06:53:543315+0000", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:10:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:10:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62050, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:10:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:10:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:10:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:10:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 17, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 17, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:10:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2096, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2146304, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 456778, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:10:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:10:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2096, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2146304, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 456778, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:10:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62050, 2024-08-30T19:10:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:10:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:10:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:10:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 185266, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 17, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 17, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-08-30T19:10:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 150323855381 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672986 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 158913789972 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:10:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:10:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:10:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:10:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 7, 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:10:43.009840+0000", 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:10:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 111, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 453, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 615, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:10:57.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:10:57.882 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-08-30T19:10:57.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-08-30T19:10:57.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:10:58.201 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:10:58.201 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 41:130 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:10:58.201 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:10:58.201 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:10:58.201 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:10:58.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:10:58.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-08-30T19:10:58.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:10:58.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:10:58.218 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:10:58.218 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:10:58.218 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:10:58.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:10:58.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:10:58.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok set_recovery_delay 100000 2024-08-30T19:10:58.330 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:10:58.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-08-30T19:10:58.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-08-30T19:10:58.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:10:58.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:10:58.345 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:10:58.345 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:10:58.346 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:10:58.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:10:58.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:10:58.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok set_recovery_delay 100000 2024-08-30T19:10:58.454 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:10:58.467 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-08-30T19:10:58.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-08-30T19:10:58.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:10:58.897 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:10:58.982 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 41:130 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:10:58.982 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:10:58.982 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:10:58.983 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:10:58.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-08-30T19:10:59.532 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-08-30T19:10:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-08-30T19:10:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:10:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:10:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:10:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:10:59.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:10:59.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:10:59.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:10:59.558 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:10:59.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:10:59.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:10:59.597 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:10:59.597+0000 7ff552df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:10:59.599 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:10:59.597+0000 7ff552df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:10:59.600 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:10:59.601+0000 7ff552df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:00.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-30T19:11:00.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-08-30T19:11:00.879 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:00.877+0000 7ff552df27c0 -1 Falling back to public interface 2024-08-30T19:11:03.892 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:03.893+0000 7ff552df27c0 -1 osd.1 28 log_to_monitors true 2024-08-30T19:11:05.014 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-08-30T19:11:05.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-08-30T19:11:05.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:05.426 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:05.426 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 43:134 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:11:05.426 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:05.426 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:05.427 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:05.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-08-30T19:11:05.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:11:05.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:11:05.443 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:11:05.443 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:05.443 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:05.443 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:11:05.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:11:05.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok set_recovery_delay 100000 2024-08-30T19:11:05.554 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:11:05.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-08-30T19:11:06.328 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-08-30T19:11:06.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-08-30T19:11:06.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:06.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:11:06.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:06.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:06.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:06.353 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:06.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:06.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:06.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:11:06.894 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 35 up_thru 40 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/353272593,v1:127.0.0.1:6803/353272593] [v2:127.0.0.1:6804/353272593,v1:127.0.0.1:6805/353272593] exists,up c984e51a-1348-4ed0-988c-566bb29481d0 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:06.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:06.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:06.897 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:06.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:06.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:11:07.342 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/3843723126,v1:127.0.0.1:6819/3843723126] [v2:127.0.0.1:6820/3843723126,v1:127.0.0.1:6821/3843723126] exists,up 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:07.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:07.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:07.344 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:07.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:07.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:11:07.787 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/2578225128,v1:127.0.0.1:6811/2578225128] [v2:127.0.0.1:6812/2578225128,v1:127.0.0.1:6813/2578225128] exists,up 21bc4db4-4fb0-41e5-9548-c0193afa3f30 2024-08-30T19:11:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:11:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:11:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:11:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:08.192 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:08.192 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 44:136 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:11:08.192 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:08.193 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:08.193 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:08.209 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-08-30T19:11:08.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-08-30T19:11:08.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:08.590 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:08.590 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 44:136 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:11:08.590 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:08.591 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:08.591 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:08.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-08-30T19:11:08.882 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-08-30T19:11:08.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-08-30T19:11:08.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:09.266 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:09.266 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-30T19:09:42.632308+0000 41'101 44:136 [0,2] 0 [0,2] 0 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:23:04.935293+0000 0 0 2024-08-30T19:11:09.266 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:09.266 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:09.266 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:09.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-08-30T19:11:09.282 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:11:09.282 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:11:09.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:11:09.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:11:09.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:11:09.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:11:09.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27261.out 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27261.out 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-30T19:11:09.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27261.out 2024-08-30T19:11:10.784 INFO:tasks.workunit.client.0.smithi079.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-08-30T19:11:10.786 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:023dd3d2:::existing_12:head# 2024-08-30T19:11:10.786 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:04c40a0c:::existing_93:head# 2024-08-30T19:11:10.787 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:064ff5fb:::existing_36:head# 2024-08-30T19:11:10.787 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:091f9f1d:::existing_94:head# 2024-08-30T19:11:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:0a865a93:::existing_54:head# 2024-08-30T19:11:10.789 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:0fed513c:::existing_41:head# 2024-08-30T19:11:10.789 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-08-30T19:11:10.790 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:169157fa:::existing_81:head# 2024-08-30T19:11:10.791 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:17731382:::existing_58:head# 2024-08-30T19:11:10.791 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:18a45a39:::existing_43:head# 2024-08-30T19:11:10.792 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:1c160f01:::existing_66:head# 2024-08-30T19:11:10.792 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:1dd6c249:::existing_97:head# 2024-08-30T19:11:10.793 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:227e0c77:::existing_7:head# 2024-08-30T19:11:10.793 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:23e1b063:::existing_59:head# 2024-08-30T19:11:10.798 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:28566edb:::existing_78:head# 2024-08-30T19:11:10.798 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:2a101182:::existing_20:head# 2024-08-30T19:11:10.798 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:2d1519da:::existing_96:head# 2024-08-30T19:11:10.798 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:3078b87a:::existing_86:head# 2024-08-30T19:11:10.798 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:30bde01b:::existing_15:head# 2024-08-30T19:11:10.799 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:317a7e19:::existing_56:head# 2024-08-30T19:11:10.799 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-08-30T19:11:10.799 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:3e16d84b:::existing_48:head# 2024-08-30T19:11:10.799 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:45fe8203:::existing_75:head# 2024-08-30T19:11:10.800 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-08-30T19:11:10.800 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5298dcdb:::existing_57:head# 2024-08-30T19:11:10.800 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:52ebaa44:::existing_32:head# 2024-08-30T19:11:10.800 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:53588912:::existing_9:head# 2024-08-30T19:11:10.800 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-08-30T19:11:10.801 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5da0255c:::existing_13:head# 2024-08-30T19:11:10.801 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5da2dab8:::existing_53:head# 2024-08-30T19:11:10.802 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5ee3514b:::existing_30:head# 2024-08-30T19:11:10.802 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:602f83fe:::foo:head# 2024-08-30T19:11:10.803 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:60426cc7:::existing_62:head# 2024-08-30T19:11:10.804 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:628918ca:::existing_79:head# 2024-08-30T19:11:10.804 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-08-30T19:11:10.805 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:67e1780f:::existing_27:head# 2024-08-30T19:11:10.805 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:68143ca1:::existing_71:head# 2024-08-30T19:11:10.805 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6bac9509:::existing_29:head# 2024-08-30T19:11:10.806 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6bbec798:::existing_45:head# 2024-08-30T19:11:10.806 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6cb23a40:::existing_65:head# 2024-08-30T19:11:10.807 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:7055ec5e:::existing_83:head# 2024-08-30T19:11:10.807 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:70a54e2b:::existing_87:head# 2024-08-30T19:11:10.808 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:750ace21:::existing_19:head# 2024-08-30T19:11:10.809 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:78623b1d:::existing_6:head# 2024-08-30T19:11:10.809 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:79881152:::existing_63:head# 2024-08-30T19:11:10.809 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-08-30T19:11:10.809 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:81490677:::existing_76:head# 2024-08-30T19:11:10.810 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-08-30T19:11:10.810 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-08-30T19:11:10.811 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-08-30T19:11:10.811 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:923c5d4d:::existing_5:head# 2024-08-30T19:11:10.812 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:92731e71:::existing_89:head# 2024-08-30T19:11:10.812 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:928205cf:::existing_82:head# 2024-08-30T19:11:10.813 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:93be2267:::existing_24:head# 2024-08-30T19:11:10.813 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:99ceda2c:::existing_44:head# 2024-08-30T19:11:10.814 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:9dd917ee:::existing_26:head# 2024-08-30T19:11:10.814 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a28805a6:::existing_72:head# 2024-08-30T19:11:10.815 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a294a8f6:::existing_16:head# 2024-08-30T19:11:10.815 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a2f5473a:::existing_42:head# 2024-08-30T19:11:10.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a504a533:::existing_90:head# 2024-08-30T19:11:10.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a691337e:::existing_61:head# 2024-08-30T19:11:10.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-08-30T19:11:10.817 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a9fde86e:::existing_18:head# 2024-08-30T19:11:10.817 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:aa309faa:::existing_68:head# 2024-08-30T19:11:10.818 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-08-30T19:11:10.818 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b279b1fe:::existing_10:head# 2024-08-30T19:11:10.819 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b343e277:::existing_88:head# 2024-08-30T19:11:10.819 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b7130e90:::existing_80:head# 2024-08-30T19:11:10.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:bc6f7309:::existing_70:head# 2024-08-30T19:11:10.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:be8489b7:::existing_38:head# 2024-08-30T19:11:10.821 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-08-30T19:11:10.821 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:bebec877:::existing_74:head# 2024-08-30T19:11:10.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c04347d8:::existing_60:head# 2024-08-30T19:11:10.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c156033e:::existing_100:head# 2024-08-30T19:11:10.823 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-08-30T19:11:10.823 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-08-30T19:11:10.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-08-30T19:11:10.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-08-30T19:11:10.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:ccb3865b:::existing_73:head# 2024-08-30T19:11:10.825 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:cd84e19e:::existing_40:head# 2024-08-30T19:11:10.825 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-08-30T19:11:10.826 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d41a25c9:::existing_22:head# 2024-08-30T19:11:10.826 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d7f088da:::existing_91:head# 2024-08-30T19:11:10.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-08-30T19:11:10.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-08-30T19:11:10.828 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd677bd3:::existing_35:head# 2024-08-30T19:11:10.828 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-08-30T19:11:10.828 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:e0e22077:::existing_21:head# 2024-08-30T19:11:10.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:e255b710:::existing_55:head# 2024-08-30T19:11:10.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:eff1d5db:::existing_14:head# 2024-08-30T19:11:10.830 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f1b156bf:::existing_23:head# 2024-08-30T19:11:10.830 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f5ce4199:::existing_34:head# 2024-08-30T19:11:10.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f6809456:::existing_11:head# 2024-08-30T19:11:10.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f699a2a9:::existing_67:head# 2024-08-30T19:11:10.832 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f9baac43:::existing_25:head# 2024-08-30T19:11:10.832 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:fb76688e:::existing_51:head# 2024-08-30T19:11:10.832 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:fbb92eec:::existing_69:head# 2024-08-30T19:11:10.833 INFO:tasks.workunit.client.0.smithi079.stderr:Export successful 2024-08-30T19:11:10.833 INFO:tasks.workunit.client.0.smithi079.stdout: marking collection for removal 2024-08-30T19:11:10.833 INFO:tasks.workunit.client.0.smithi079.stdout:setting '_remove' omap key 2024-08-30T19:11:10.833 INFO:tasks.workunit.client.0.smithi079.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-08-30T19:11:10.992 INFO:tasks.workunit.client.0.smithi079.stderr:Remove successful 2024-08-30T19:11:11.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27261.out 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-30T19:11:11.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27261.out 2024-08-30T19:11:12.532 INFO:tasks.workunit.client.0.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-08-30T19:11:12.532 INFO:tasks.workunit.client.0.smithi079.stdout:Importing pgid 1.0 2024-08-30T19:11:12.533 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:023dd3d2:::existing_12:head# 2024-08-30T19:11:12.533 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.596 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:04c40a0c:::existing_93:head# 2024-08-30T19:11:12.596 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.629 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:064ff5fb:::existing_36:head# 2024-08-30T19:11:12.629 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.663 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:091f9f1d:::existing_94:head# 2024-08-30T19:11:12.663 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.714 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:0a865a93:::existing_54:head# 2024-08-30T19:11:12.714 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.746 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:0fed513c:::existing_41:head# 2024-08-30T19:11:12.746 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.780 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-08-30T19:11:12.780 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.813 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:169157fa:::existing_81:head# 2024-08-30T19:11:12.813 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.847 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:17731382:::existing_58:head# 2024-08-30T19:11:12.847 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.880 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:18a45a39:::existing_43:head# 2024-08-30T19:11:12.880 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.913 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:1c160f01:::existing_66:head# 2024-08-30T19:11:12.914 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.947 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:1dd6c249:::existing_97:head# 2024-08-30T19:11:12.947 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.972 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:227e0c77:::existing_7:head# 2024-08-30T19:11:12.972 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:12.997 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:23e1b063:::existing_59:head# 2024-08-30T19:11:12.997 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.022 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:28566edb:::existing_78:head# 2024-08-30T19:11:13.022 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.048 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:2a101182:::existing_20:head# 2024-08-30T19:11:13.048 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.073 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:2d1519da:::existing_96:head# 2024-08-30T19:11:13.073 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.139 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:3078b87a:::existing_86:head# 2024-08-30T19:11:13.140 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.190 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:30bde01b:::existing_15:head# 2024-08-30T19:11:13.190 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.215 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:317a7e19:::existing_56:head# 2024-08-30T19:11:13.215 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.240 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-08-30T19:11:13.240 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.265 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:3e16d84b:::existing_48:head# 2024-08-30T19:11:13.265 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.290 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:45fe8203:::existing_75:head# 2024-08-30T19:11:13.290 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.315 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-08-30T19:11:13.315 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.340 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5298dcdb:::existing_57:head# 2024-08-30T19:11:13.341 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.365 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:52ebaa44:::existing_32:head# 2024-08-30T19:11:13.366 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.391 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:53588912:::existing_9:head# 2024-08-30T19:11:13.391 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.416 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-08-30T19:11:13.416 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.441 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5da0255c:::existing_13:head# 2024-08-30T19:11:13.441 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.466 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5da2dab8:::existing_53:head# 2024-08-30T19:11:13.466 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.491 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5ee3514b:::existing_30:head# 2024-08-30T19:11:13.491 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.516 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:602f83fe:::foo:head# 2024-08-30T19:11:13.516 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.541 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:60426cc7:::existing_62:head# 2024-08-30T19:11:13.542 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.608 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:628918ca:::existing_79:head# 2024-08-30T19:11:13.609 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.659 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-08-30T19:11:13.659 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.701 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:67e1780f:::existing_27:head# 2024-08-30T19:11:13.701 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.734 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:68143ca1:::existing_71:head# 2024-08-30T19:11:13.734 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.767 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6bac9509:::existing_29:head# 2024-08-30T19:11:13.768 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.801 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6bbec798:::existing_45:head# 2024-08-30T19:11:13.801 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.834 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6cb23a40:::existing_65:head# 2024-08-30T19:11:13.835 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.869 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:7055ec5e:::existing_83:head# 2024-08-30T19:11:13.869 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.901 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:70a54e2b:::existing_87:head# 2024-08-30T19:11:13.901 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.935 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:750ace21:::existing_19:head# 2024-08-30T19:11:13.935 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:13.968 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:78623b1d:::existing_6:head# 2024-08-30T19:11:13.968 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.002 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:79881152:::existing_63:head# 2024-08-30T19:11:14.002 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.035 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-08-30T19:11:14.035 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.072 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:81490677:::existing_76:head# 2024-08-30T19:11:14.072 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.102 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-08-30T19:11:14.102 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.135 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-08-30T19:11:14.136 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.169 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-08-30T19:11:14.169 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.211 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:923c5d4d:::existing_5:head# 2024-08-30T19:11:14.211 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.244 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:92731e71:::existing_89:head# 2024-08-30T19:11:14.245 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.278 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:928205cf:::existing_82:head# 2024-08-30T19:11:14.278 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.312 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:93be2267:::existing_24:head# 2024-08-30T19:11:14.312 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.345 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:99ceda2c:::existing_44:head# 2024-08-30T19:11:14.345 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.378 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:9dd917ee:::existing_26:head# 2024-08-30T19:11:14.379 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.412 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a28805a6:::existing_72:head# 2024-08-30T19:11:14.412 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.445 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a294a8f6:::existing_16:head# 2024-08-30T19:11:14.445 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.481 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a2f5473a:::existing_42:head# 2024-08-30T19:11:14.481 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.512 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a504a533:::existing_90:head# 2024-08-30T19:11:14.512 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.546 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a691337e:::existing_61:head# 2024-08-30T19:11:14.546 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.579 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-08-30T19:11:14.579 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.613 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a9fde86e:::existing_18:head# 2024-08-30T19:11:14.613 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.646 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:aa309faa:::existing_68:head# 2024-08-30T19:11:14.646 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.679 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-08-30T19:11:14.680 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.738 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b279b1fe:::existing_10:head# 2024-08-30T19:11:14.738 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.772 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b343e277:::existing_88:head# 2024-08-30T19:11:14.772 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.806 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b7130e90:::existing_80:head# 2024-08-30T19:11:14.806 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.838 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:bc6f7309:::existing_70:head# 2024-08-30T19:11:14.839 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.872 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:be8489b7:::existing_38:head# 2024-08-30T19:11:14.872 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.905 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-08-30T19:11:14.905 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.939 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:bebec877:::existing_74:head# 2024-08-30T19:11:14.939 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:14.972 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c04347d8:::existing_60:head# 2024-08-30T19:11:14.972 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.006 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c156033e:::existing_100:head# 2024-08-30T19:11:15.006 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.039 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-08-30T19:11:15.039 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.073 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-08-30T19:11:15.073 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.131 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-08-30T19:11:15.131 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.156 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-08-30T19:11:15.157 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.181 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:ccb3865b:::existing_73:head# 2024-08-30T19:11:15.182 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.207 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:cd84e19e:::existing_40:head# 2024-08-30T19:11:15.207 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.232 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-08-30T19:11:15.232 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.257 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d41a25c9:::existing_22:head# 2024-08-30T19:11:15.257 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.282 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d7f088da:::existing_91:head# 2024-08-30T19:11:15.282 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.307 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-08-30T19:11:15.307 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.332 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-08-30T19:11:15.332 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.357 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd677bd3:::existing_35:head# 2024-08-30T19:11:15.357 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.382 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-08-30T19:11:15.383 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.416 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:e0e22077:::existing_21:head# 2024-08-30T19:11:15.416 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.449 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:e255b710:::existing_55:head# 2024-08-30T19:11:15.449 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.474 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:eff1d5db:::existing_14:head# 2024-08-30T19:11:15.475 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.500 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f1b156bf:::existing_23:head# 2024-08-30T19:11:15.500 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.525 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f5ce4199:::existing_34:head# 2024-08-30T19:11:15.525 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.559 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f6809456:::existing_11:head# 2024-08-30T19:11:15.559 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.634 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f699a2a9:::existing_67:head# 2024-08-30T19:11:15.634 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.701 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f9baac43:::existing_25:head# 2024-08-30T19:11:15.701 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.726 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:fb76688e:::existing_51:head# 2024-08-30T19:11:15.726 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.751 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:fbb92eec:::existing_69:head# 2024-08-30T19:11:15.751 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:11:15.776 INFO:tasks.workunit.client.0.smithi079.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (28'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-08-30T19:11:15.809 INFO:tasks.workunit.client.0.smithi079.stdout:Import successful 2024-08-30T19:11:16.347 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-08-30T19:11:16.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-08-30T19:11:16.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:16.755 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:16.755 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-08-30T19:11:07.775470+0000 47'102 47:134 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 no scrub is scheduled 0 0 2024-08-30T19:11:16.755 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:16.755 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:16.756 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:11:16.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:11:16.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:11:16.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:11:16.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:11:16.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:11:16.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:11:16.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:11:16.776 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:11:16.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:11:16.810 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:16.808+0000 7fe8089037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:16.812 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:16.812+0000 7fe8089037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:16.813 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:16.812+0000 7fe8089037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:17.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:11:17.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:17.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:11:17.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:17.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:11:17.584 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/3843723126,v1:127.0.0.1:6819/3843723126] [v2:127.0.0.1:6820/3843723126,v1:127.0.0.1:6821/3843723126] exists,up 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:17.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:17.587 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:17.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:17.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:11:17.870 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:17.868+0000 7fe8089037c0 -1 Falling back to public interface 2024-08-30T19:11:18.024 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/3843723126,v1:127.0.0.1:6819/3843723126] [v2:127.0.0.1:6820/3843723126,v1:127.0.0.1:6821/3843723126] exists,up 0b137e2a-5f55-4bf4-885d-48c5c7c60eca 2024-08-30T19:11:18.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:11:18.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:11:18.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:11:18.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-08-30T19:11:20.751 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:20.748+0000 7fe8089037c0 -1 osd.1 47 log_to_monitors true 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:38.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:38.028 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:11:38.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-08-30T19:11:38.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok dump_ops_in_flight 2024-08-30T19:11:38.145 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:11:38.145 INFO:tasks.workunit.client.0.smithi079.stdout: "ops": [], 2024-08-30T19:11:38.145 INFO:tasks.workunit.client.0.smithi079.stdout: "num_ops": 0 2024-08-30T19:11:38.145 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:11:38.157 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-08-30T19:11:38.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-08-30T19:11:38.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:38.562 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:38.562 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-08-30T19:11:22.330229+0000 47'102 51:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:11:14.582794+0000 0 0 2024-08-30T19:11:38.562 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:38.562 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:38.562 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:38.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-30T19:11:38.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-30T19:11:38.728 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:11:38.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-30T19:11:38.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-30T19:11:38.894 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:11:38.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-30T19:11:38.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-30T19:11:39.057 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:11:39.072 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-08-30T19:11:39.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-08-30T19:11:39.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-08-30T19:11:39.456 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:11:39.456 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-08-30T19:11:22.330229+0000 47'102 51:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:08:42.880661+0000 0'0 2024-08-30T19:08:42.880661+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:11:14.582794+0000 0 0 2024-08-30T19:11:39.457 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:11:39.457 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:11:39.457 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:11:39.473 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-08-30T19:11:39.473 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-08-30T19:11:39.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:11:39.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-08-30T19:11:39.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:11:39.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-08-30T19:11:39.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:11:39.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-08-30T19:11:39.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:11:39.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-08-30T19:11:39.674 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-08-30T19:11:39.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:11:39.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-08-30T19:11:39.709 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_1: (2) No such file or directory 2024-08-30T19:11:39.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:11:39.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-08-30T19:11:39.743 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_2: (2) No such file or directory 2024-08-30T19:11:39.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-08-30T19:11:39.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-30T19:11:39.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:11:39.751 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4639.0:1 2024-08-30T19:09:16.641258+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4639.0:1 2024-08-30T19:09:16.641258+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-08-30T19:10:20.401775+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4626.0:1 2024-08-30T19:09:16.617348+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4626.0:1 2024-08-30T19:09:16.617348+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-30T19:11:39.752 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-08-30T19:09:16.636454+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-08-30T19:09:16.636454+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'104 last_divergent_update: 28'104 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'104 (28'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-08-30T19:09:16.636454+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-08-30T19:09:16.621839+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-08-30T19:09:16.621839+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.753 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'102 last_divergent_update: 28'102 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-08-30T19:09:16.621839+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4637.0:1 2024-08-30T19:09:16.638138+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.754 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4637.0:1 2024-08-30T19:09:16.638138+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'105 last_divergent_update: 28'105 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'105 (28'1) modify 1:d858c035:::existing_1:head by client.4637.0:1 2024-08-30T19:09:16.638138+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4632.0:1 2024-08-30T19:09:16.631643+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4632.0:1 2024-08-30T19:09:16.631643+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2024-08-30T19:11:39.755 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-08-30T19:11:39.756 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:11:07.661+0000 7ff536d7c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-08-30T19:11:39.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-30T19:11:39.756 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:11:39.759 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-30T19:11:39.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:11:39.764 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-08-30T19:11:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-08-30T19:11:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27261.out 2024-08-30T19:11:39.765 INFO:tasks.workunit.client.0.smithi079.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27261.out': No such file or directory 2024-08-30T19:11:39.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-08-30T19:11:39.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-30T19:11:39.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-30T19:11:40.244 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-08-30T19:11:40.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-08-30T19:11:40.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:11:40.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:11:40.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:11:40.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:11:40.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:11:45.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:11:45.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-30T19:11:45.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:11:45.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:11:45.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:11:45.580 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:11:45.580 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:11:45.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:11:45.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:11:45.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:11:45.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:11:45.591 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:11:45.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:11:45.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:11:45.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:11:45.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:11:45.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:11:45.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:11:45.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:11:45.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:11:45.596 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:11:45.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:11:45.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:11:45.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:11:45.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:11:45.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:45.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:45.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-30T19:11:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:11:45.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:11:45.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:11:45.634 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:11:45.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:11:45.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:11:45.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:11:45.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:11:45.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:11:45.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:11:45.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:11:45.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:11:45.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:11:45.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:11:45.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:11:45.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:11:45.642 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:11:45.642 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:45.642 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:45.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:11:45.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:11:45.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:11:45.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-30T19:11:45.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:11:45.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:45.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:45.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27261 2024-08-30T19:11:45.647 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:11:45.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-08-30T19:11:45.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-08-30T19:11:45.649 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-08-30T19:11:45.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-08-30T19:11:45.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:11:45.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-30T19:11:45.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-30T19:11:46.016 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:11:46.016 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:11:46.016 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:11:46.016 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:11:46.017 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:46.017 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:46.020 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:46.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:11:46.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:11:46.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:11:46.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:11:46.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:11:46.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:11:46.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:11:46.229 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:11:46.230 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:11:46.230 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:11:46.230 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:11:46.230 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:46.230 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:46.231 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:11:46.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:11:46.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get fsid 2024-08-30T19:11:46.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:11:46.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:11:46.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:11:46.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:11:46.347 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:11:46.347 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:11:46.347 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:11:46.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:11:46.347 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:11:46.348 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:46.348 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:46.348 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:11:46.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:11:46.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get mon_host 2024-08-30T19:11:46.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-30T19:11:46.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:11:47.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:11:47.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:11:47.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:11:47.356 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:11:47.357 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:47.357 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:47.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:47.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:11:47.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:11:47.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-30T19:11:47.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-08-30T19:11:47.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:11:47.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:11:47.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:11:47.400 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:11:47.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f556141b-5721-4aa9-9831-2ea62c079268' 2024-08-30T19:11:47.400 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:11:47.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzGdJmyi1eGRAAcCNKmCUwbJK6gR96FOc5mg== 2024-08-30T19:11:47.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzGdJmyi1eGRAAcCNKmCUwbJK6gR96FOc5mg=="}' 2024-08-30T19:11:47.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f556141b-5721-4aa9-9831-2ea62c079268 -i td/divergent-priors/0/new.json 2024-08-30T19:11:47.748 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:47.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-30T19:11:47.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzGdJmyi1eGRAAcCNKmCUwbJK6gR96FOc5mg== --osd-uuid f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:11:47.801 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:47.800+0000 7ff41b70a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:47.816 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:47.816+0000 7ff41b70a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:47.817 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:47.816+0000 7ff41b70a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:47.818 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:47.816+0000 7ff41b70a7c0 -1 bdev(0x564d6f978a80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:11:47.818 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:47.816+0000 7ff41b70a7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-30T19:11:54.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-30T19:11:54.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:11:54.410 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:11:54.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:11:54.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:11:54.991 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:11:54.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:11:54.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:11:54.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:11:54.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:11:54.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:11:55.031 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:55.032+0000 7f2c52ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:55.054 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:55.056+0000 7f2c52ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:55.055 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:55.056+0000 7f2c52ebe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:11:55.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:11:55.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:11:55.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:11:55.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:11:55.417 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:11:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:11:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:11:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:55.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:11:55.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:11:56.070 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:56.072+0000 7f2c52ebe7c0 -1 Falling back to public interface 2024-08-30T19:11:56.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:11:56.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:56.885 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:11:56.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:11:56.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:56.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:11:57.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:11:58.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:11:58.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:58.312 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:11:58.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:11:58.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:58.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:11:58.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:11:58.900 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:11:58.904+0000 7f2c52ebe7c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:11:59.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:11:59.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:11:59.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:11:59.726 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:11:59.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:11:59.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:00.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:01.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:01.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:01.244 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:12:01.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:12:01.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:01.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:01.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:02.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:02.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:02.835 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:12:02.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:12:02.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:02.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:03.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:04.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:04.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:04.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:12:04.305 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:12:04.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:04.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:04.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:05.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:05.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:05.883 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:12:05.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:12:05.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:05.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:06.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:07.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:07.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:07.304 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:12:07.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:12:07.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:07.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:07.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:08.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:08.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:08.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:12:08.858 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:12:08.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:08.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:09.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:10.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:10.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:10.297 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:12:10.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:12:10.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:10.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:10.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:11.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:11.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:11.802 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:12:11.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:12:11.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:11.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:12.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:13.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:13.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:13.517 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:12:13.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:12:13.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:13.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:14.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:15.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:15.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:15.137 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:12:15.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:12:15.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:15.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:15.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:16.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:16.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:16.955 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:12:16.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:12:16.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:16.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:17.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:18.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:18.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:18.793 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:12:18.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:12:18.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:18.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:19.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:20.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:20.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:20.566 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:12:20.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:12:20.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:20.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:21.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:22.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:22.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:22.253 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:12:22.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:12:22.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:22.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:23.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:24.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:24.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:24.445 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:12:24.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:12:24.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:24.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:12:24.916 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1708408575,v1:127.0.0.1:6803/1708408575] [v2:127.0.0.1:6804/1708408575,v1:127.0.0.1:6805/1708408575] exists,up f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:12:24.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:12:24.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:12:24.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:12:24.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:12:24.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:12:24.923 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:12:24.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9d444242-ce4b-4396-b69c-98884ed22ebf' 2024-08-30T19:12:24.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:12:24.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYGdJmE2/LOBAA7SqSrvgaN26eJ4ZjmNT6/g== 2024-08-30T19:12:24.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYGdJmE2/LOBAA7SqSrvgaN26eJ4ZjmNT6/g=="}' 2024-08-30T19:12:24.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d444242-ce4b-4396-b69c-98884ed22ebf -i td/divergent-priors/1/new.json 2024-08-30T19:12:25.518 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:12:25.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-30T19:12:25.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYGdJmE2/LOBAA7SqSrvgaN26eJ4ZjmNT6/g== --osd-uuid 9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:12:25.579 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:25.583+0000 7f41178187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:25.591 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:25.595+0000 7f41178187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:25.593 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:25.595+0000 7f41178187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:25.593 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:25.595+0000 7f41178187c0 -1 bdev(0x5573ff8eea80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:12:25.593 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:25.595+0000 7f41178187c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-30T19:12:32.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-30T19:12:32.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:12:32.034 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:12:32.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:12:32.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:12:32.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:12:32.591 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:12:32.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:12:32.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:12:32.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:12:32.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:12:32.633 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:32.635+0000 7f94fd76a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:32.646 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:32.647+0000 7f94fd76a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:32.647 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:32.651+0000 7f94fd76a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:33.022 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:12:33.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:12:33.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:33.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:33.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:33.886 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:33.887+0000 7f94fd76a7c0 -1 Falling back to public interface 2024-08-30T19:12:34.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:34.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:34.456 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:12:34.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:12:34.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:34.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:34.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:35.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:35.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:35.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:12:35.869 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:12:35.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:35.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:36.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:36.906 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:12:36.907+0000 7f94fd76a7c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:12:37.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:37.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:37.291 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:12:37.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:12:37.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:37.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:38.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:38.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:38.801 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:12:38.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:12:38.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:38.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:39.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:40.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:40.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:40.271 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:12:40.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:12:40.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:40.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:40.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:41.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:41.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:41.782 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:12:41.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:12:41.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:41.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:42.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:43.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:43.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:43.266 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:12:43.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:12:43.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:43.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:43.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:44.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:44.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:44.840 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:12:44.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:12:44.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:44.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:45.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:46.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:46.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:46.303 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:12:46.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:12:46.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:46.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:46.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:47.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:47.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:47.903 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:12:47.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:12:47.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:47.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:48.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:49.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:49.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:49.369 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:12:49.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:12:49.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:49.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:49.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:50.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:50.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:50.967 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:12:50.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:12:50.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:50.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:51.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:52.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:52.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:52.761 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:12:52.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:12:52.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:52.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:53.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:54.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:54.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:54.739 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:12:54.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:12:54.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:54.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:55.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:56.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:56.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:56.591 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:12:56.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:12:56.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:56.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:57.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:12:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:12:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:12:58.775 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:12:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:12:58.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:12:58.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:12:59.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:00.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:00.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:00.634 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:13:00.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:13:00.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:00.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:13:01.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:02.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:02.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:02.576 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:13:02.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:13:02.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:02.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:13:03.003 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4157510201,v1:127.0.0.1:6811/4157510201] [v2:127.0.0.1:6812/4157510201,v1:127.0.0.1:6813/4157510201] exists,up 9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:13:03.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:13:03.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:13:03.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:13:03.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e97e52bb-639a-403e-90ba-53ab43e5f83a' 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:13:03.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:13:03.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/GdJmESkXAhAASkPXIH/S+F6lR/p0c8VJMg== 2024-08-30T19:13:03.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/GdJmESkXAhAASkPXIH/S+F6lR/p0c8VJMg=="}' 2024-08-30T19:13:03.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e97e52bb-639a-403e-90ba-53ab43e5f83a -i td/divergent-priors/2/new.json 2024-08-30T19:13:03.651 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:13:03.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-30T19:13:03.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/GdJmESkXAhAASkPXIH/S+F6lR/p0c8VJMg== --osd-uuid e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:13:03.712 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:03.715+0000 7f5b18e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:03.735+0000 7f5b18e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:03.735+0000 7f5b18e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:03.735+0000 7f5b18e887c0 -1 bdev(0x561b7769ea80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:13:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:03.735+0000 7f5b18e887c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-30T19:13:10.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-30T19:13:10.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:13:10.719 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:13:10.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:13:10.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:13:11.281 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:13:11.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:13:11.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:13:11.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:13:11.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:13:11.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:13:11.323 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:11.327+0000 7f805fb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:11.353 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:11.355+0000 7f805fb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:11.354 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:11.355+0000 7f805fb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:13:11.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:13:11.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:13:11.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:13:11.726 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:11.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:12.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:12.837 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:12.839+0000 7f805fb407c0 -1 Falling back to public interface 2024-08-30T19:13:13.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:13.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:13.160 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:13:13.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:13:13.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:13.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:13.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:14.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:14.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:14.576 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:13:14.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:13:14.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:14.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:15.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:15.801 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:13:15.803+0000 7f805fb407c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:13:16.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:16.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:16.018 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:13:16.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:13:16.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:16.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:16.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:17.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:17.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:17.588 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:13:17.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:13:17.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:17.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:18.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:13:19.047 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:13:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:19.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:20.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:20.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:20.488 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:13:20.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:13:20.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:20.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:21.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:22.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:22.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:22.031 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:13:22.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:13:22.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:22.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:22.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:23.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:23.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:23.470 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:13:23.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:13:23.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:23.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:24.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:25.051 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:13:25.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:13:25.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:25.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:25.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:26.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:26.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:26.507 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:13:26.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:13:26.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:26.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:27.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:28.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:28.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:28.045 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:13:28.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:13:28.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:28.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:28.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:29.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:29.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:29.480 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:13:29.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:13:29.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:29.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:30.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:31.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:31.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:31.161 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:13:31.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:13:31.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:31.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:32.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:33.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:33.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:33.167 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:13:33.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:13:33.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:33.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:34.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:35.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:35.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:35.099 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:13:35.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:13:35.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:35.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:35.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:36.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:36.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:36.867 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:13:36.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:13:36.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:36.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:37.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:38.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:38.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:38.807 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:13:38.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:13:38.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:38.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:40.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:13:41.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:13:41.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:13:41.021 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:13:41.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:13:41.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:13:41.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:13:41.509 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3865108371,v1:127.0.0.1:6819/3865108371] [v2:127.0.0.1:6820/3865108371,v1:127.0.0.1:6821/3865108371] exists,up e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:13:41.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:13:41.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:13:41.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:13:41.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-08-30T19:13:42.448 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-08-30T19:13:42.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-08-30T19:13:43.277 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-08-30T19:13:43.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-08-30T19:13:44.163 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-08-30T19:13:44.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-08-30T19:13:44.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-30T19:13:44.806 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-08-30T19:13:44.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:13:45.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-08-30T19:13:47.083 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-08-30T19:13:47.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-08-30T19:13:47.827 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-08-30T19:13:47.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-08-30T19:13:48.673 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 pg_autoscale_mode to on 2024-08-30T19:13:48.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-08-30T19:13:48.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-08-30T19:13:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1725045228 2024-08-30T19:13:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-08-30T19:13:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-08-30T19:13:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-08-30T19:13:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:13:48.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:13:49.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:13:49.228 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:13:49.228 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:13:49.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:13:49.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:49.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:13:49.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-30T19:13:49.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-30T19:13:49.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-30T19:13:49.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:49.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:13:49.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-08-30T19:13:49.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-08-30T19:13:49.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975' 2024-08-30T19:13:49.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:49.566 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:13:49.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-08-30T19:13:49.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-08-30T19:13:49.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975 2-64424509449' 2024-08-30T19:13:49.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:49.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-30T19:13:49.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:49.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:13:49.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-30T19:13:49.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-30T19:13:49.725 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836505 2024-08-30T19:13:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-30T19:13:49.726 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:13:50.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-30T19:13:50.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:50.167 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:50.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-08-30T19:13:50.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:13:50.169 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-08-30T19:13:50.170 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:50.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-08-30T19:13:50.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-08-30T19:13:50.171 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672975 2024-08-30T19:13:50.171 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:13:50.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-08-30T19:13:50.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:50.599 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-08-30T19:13:50.599 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:50.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:13:50.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-08-30T19:13:50.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:50.602 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-08-30T19:13:50.603 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509449 2024-08-30T19:13:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-08-30T19:13:50.603 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:13:51.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-08-30T19:13:51.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-08-30T19:13:51.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:13:51.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:13:51.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:13:51.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:13:51.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:13:51.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:13:51.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:13:51.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:13:51.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:13:51.123 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:13:51.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:13:51.555 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:13:51.555 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:13:51.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:13:51.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:51.555 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:13:51.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-08-30T19:13:51.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-08-30T19:13:51.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-08-30T19:13:51.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:51.726 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:13:51.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-30T19:13:51.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-30T19:13:51.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977' 2024-08-30T19:13:51.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:13:51.884 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:13:52.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-08-30T19:13:52.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-08-30T19:13:52.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509452' 2024-08-30T19:13:52.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:52.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-08-30T19:13:52.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:52.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:13:52.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-08-30T19:13:52.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:52.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-08-30T19:13:52.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-08-30T19:13:52.048 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836508 2024-08-30T19:13:52.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:13:52.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-08-30T19:13:52.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:52.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-30T19:13:52.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:52.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:13:52.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-30T19:13:52.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:52.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-30T19:13:52.552 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672977 2024-08-30T19:13:52.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-30T19:13:52.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:13:53.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-30T19:13:53.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:13:53.008 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-08-30T19:13:53.008 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:13:53.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:13:53.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-08-30T19:13:53.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:13:53.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-08-30T19:13:53.012 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509452 2024-08-30T19:13:53.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-08-30T19:13:53.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:13:53.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509452 2024-08-30T19:13:53.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:13:54.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:13:54.468 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:13:55.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-08-30T19:13:55.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:13:55.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:13:55.070 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:13:55.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:13:55.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:13:55.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:13:55.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:13:55.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:13:55.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:13:55.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:13:55.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:13:56.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:13:56.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:13:56.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:13:56.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:13:56.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 32 2024-08-30T19:13:56.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-08-30T19:13:56.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:13:56.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-08-30T19:13:56.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:13:56.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:13:56.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:13:56.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:13:56.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:13:56.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:13:56.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:13:56.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:13:56.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:13:56.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:13:57.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:13:57.199 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:13:57.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:13:57.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:13:57.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 32 2024-08-30T19:13:57.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:13:57.765 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:13:58.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:13:58.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:13:58.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-30T19:13:58.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:13:58.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:13:58.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:13:58.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:13:58.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:13:58.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:13:58.933 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:13:58.934 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:13:58.934 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:13:59.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 32 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:13:59.500 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:13:59.501 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:14:00.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:14:00.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:14:00.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-30T19:14:00.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:14:00.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:14:00.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:14:00.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:14:00.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:14:00.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:14:00.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:14:00.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:14:00.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:14:00.582 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:14:01.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-30T19:14:01.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:14:01.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:01.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 1 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=24 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:14:01.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:14:01.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:14:01.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:14:01.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:14:01.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:14:01.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:14:01.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:14:01.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:14:01.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:14:02.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-30T19:14:02.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:14:02.367 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:02.367 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:02.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 24 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:14:02.945 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:14:02.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:14:03.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:14:03.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:14:03.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-30T19:14:03.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:14:03.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:14:03.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:14:03.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:14:03.782 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:14:03.782 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:14:03.782 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:14:03.783 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:14:03.783 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:14:03.783 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:14:04.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-30T19:14:04.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:14:04.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:04.219 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 24 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:14:04.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:14:04.820 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:14:04.820 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:14:05.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:14:05.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:14:05.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-30T19:14:05.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:14:05.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:14:05.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:14:05.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:14:05.858 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:14:05.858 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:14:05.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:14:05.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:14:05.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:14:05.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:14:06.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-30T19:14:06.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:14:06.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:06.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:06.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-30T19:14:06.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:14:06.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:14:06.828 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-08-30T19:14:06.828 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-08-30T19:14:07.264 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:14:07.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=0 2024-08-30T19:14:07.282 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 0 2024-08-30T19:14:07.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 0' 2024-08-30T19:14:07.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:14:07.670 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:14:07.670 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:03.342640+0000 0'0 37:49 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:49:32.281029+0000 0 0 2024-08-30T19:14:07.670 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.967431+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:01:01.995704+0000 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.196291+0000 0'0 37:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.200171+0000 0'0 37:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.991679+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.993416+0000 0'0 37:57 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:14:43.023575+0000 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.994553+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.994100+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.197340+0000 0'0 37:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.213165+0000 0'0 36:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:56.998645+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:56:06.197361+0000 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.994336+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.671 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.995929+0000 0'0 37:41 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.202220+0000 0'0 37:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.211667+0000 0'0 36:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:13:58.946608+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.994054+0000 0'0 37:41 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.981254+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:32:44.020331+0000 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.211516+0000 0'0 36:39 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.993630+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.212584+0000 0'0 36:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:02.371815+0000 0'0 37:49 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:21:14.107565+0000 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.996148+0000 0'0 37:41 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:05.297331+0000 0'0 37:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:35:28.951861+0000 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:57.955314+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:55:16.167653+0000 0 0 2024-08-30T19:14:07.672 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.213667+0000 0'0 36:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:13:55.198000+0000 0'0 37:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:00.972852+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:53.256747+0000 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:04.324535+0000 0'0 37:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:53:41.733485+0000 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.211051+0000 0'0 36:40 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:02.930167+0000 0'0 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:08:46.610741+0000 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.993272+0000 0'0 37:49 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:14:01.996361+0000 0'0 37:41 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:13:44.640725+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 queued for scrub 0 0 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:14:07.673 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:14:07.686 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-08-30T19:14:07.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1725045247 2024-08-30T19:14:07.687 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1725045247 - 1725045228 2024-08-30T19:14:07.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=19 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 19 -gt 300 ']' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 0 -le -1 ']' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 0 ']' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 0 ']' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 0 ']' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1 2' 2024-08-30T19:14:07.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-08-30T19:14:07.691 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-08-30T19:14:07.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-08-30T19:14:07.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:07.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-08-30T19:14:07.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:07.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-08-30T19:14:08.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:08.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-08-30T19:14:08.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:08.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-08-30T19:14:08.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:08.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-08-30T19:14:09.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:09.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-08-30T19:14:09.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:09.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-08-30T19:14:09.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:09.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-08-30T19:14:09.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:09.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-08-30T19:14:09.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:09.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-08-30T19:14:10.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:10.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-08-30T19:14:10.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:10.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-08-30T19:14:10.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:10.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-08-30T19:14:10.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:10.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-08-30T19:14:10.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:10.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-08-30T19:14:11.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:11.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-08-30T19:14:11.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:11.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-08-30T19:14:11.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:11.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-08-30T19:14:11.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:11.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-08-30T19:14:11.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:11.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-08-30T19:14:12.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:12.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-08-30T19:14:12.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:12.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-08-30T19:14:12.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:12.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-08-30T19:14:12.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:12.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-08-30T19:14:12.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:12.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-08-30T19:14:13.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:13.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-08-30T19:14:13.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:13.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-08-30T19:14:13.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:13.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-08-30T19:14:13.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:13.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-08-30T19:14:13.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:13.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-08-30T19:14:14.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:14.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-08-30T19:14:14.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:14.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-08-30T19:14:14.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:14.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-08-30T19:14:14.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:14.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-08-30T19:14:14.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:14.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-08-30T19:14:15.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:15.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-08-30T19:14:15.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:15.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-08-30T19:14:15.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:15.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-08-30T19:14:15.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:15.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-08-30T19:14:15.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:15.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-08-30T19:14:16.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-08-30T19:14:16.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-08-30T19:14:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-08-30T19:14:16.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-08-30T19:14:16.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-08-30T19:14:16.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:16.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-08-30T19:14:17.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-08-30T19:14:17.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-08-30T19:14:17.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-08-30T19:14:17.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-08-30T19:14:17.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-08-30T19:14:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:17.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-08-30T19:14:18.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-08-30T19:14:18.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-08-30T19:14:18.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-08-30T19:14:18.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-08-30T19:14:18.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-08-30T19:14:18.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:18.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-08-30T19:14:19.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-08-30T19:14:19.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-08-30T19:14:19.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-08-30T19:14:19.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-08-30T19:14:19.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-08-30T19:14:19.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:19.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-08-30T19:14:20.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-08-30T19:14:20.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-08-30T19:14:20.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-08-30T19:14:20.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-08-30T19:14:20.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-08-30T19:14:20.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:20.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-08-30T19:14:21.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:21.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-08-30T19:14:21.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:21.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-08-30T19:14:21.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:21.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-08-30T19:14:21.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:21.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-08-30T19:14:21.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:21.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-08-30T19:14:22.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-08-30T19:14:22.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-08-30T19:14:22.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-08-30T19:14:22.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-08-30T19:14:22.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-08-30T19:14:22.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:22.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-08-30T19:14:23.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-08-30T19:14:23.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-08-30T19:14:23.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-08-30T19:14:23.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-08-30T19:14:23.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-08-30T19:14:23.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:23.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-08-30T19:14:24.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:24.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-08-30T19:14:24.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:24.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-08-30T19:14:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-08-30T19:14:24.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:24.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-08-30T19:14:24.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:24.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-08-30T19:14:25.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-08-30T19:14:25.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-08-30T19:14:25.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-08-30T19:14:25.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-08-30T19:14:25.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-08-30T19:14:25.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:25.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-08-30T19:14:26.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:26.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-08-30T19:14:26.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-30T19:14:26.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-08-30T19:14:26.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:14:26.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-08-30T19:14:26.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:14:26.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:14:26.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:14:26.459 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:14:26.459 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:14:26.459 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:14:26.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:14:26.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:14:26.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:14:26.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:14:26.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:14:26.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:14:26.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:14:26.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:14:26.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:14:26.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:14:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:14:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:14:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:14:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:26.975 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:14:27.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-08-30T19:14:27.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-08-30T19:14:27.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-08-30T19:14:27.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:27.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:14:27.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-08-30T19:14:27.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-08-30T19:14:27.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672986' 2024-08-30T19:14:27.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:27.319 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:14:27.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-08-30T19:14:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-08-30T19:14:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672986 2-64424509461' 2024-08-30T19:14:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:14:27.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-08-30T19:14:27.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:14:27.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:14:27.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-08-30T19:14:27.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:14:27.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-08-30T19:14:27.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-08-30T19:14:27.485 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836517 2024-08-30T19:14:27.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:14:27.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836517 2024-08-30T19:14:27.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:14:28.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:14:28.914 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:14:29.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836517 2024-08-30T19:14:29.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:14:29.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-08-30T19:14:29.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:14:29.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:14:29.353 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-08-30T19:14:29.353 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:14:29.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-08-30T19:14:29.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-08-30T19:14:29.355 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672986 2024-08-30T19:14:29.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:14:29.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-08-30T19:14:29.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:14:29.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-08-30T19:14:29.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:14:29.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:14:29.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-08-30T19:14:29.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:14:29.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-08-30T19:14:29.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-08-30T19:14:29.800 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509461 2024-08-30T19:14:29.801 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:14:30.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-08-30T19:14:30.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:14:30.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:30.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:30.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:14:30.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:14:31.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-30T19:14:31.253 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:14:31.254 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:14:31.254 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:14:31.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-30T19:14:31.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:14:31.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:14:31.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-08-30T19:14:31.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-30T19:14:31.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-30T19:14:31.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-30T19:14:31.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-30T19:14:32.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-08-30T19:14:32.218 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 1 2 2024-08-30T19:14:32.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 1 2' 2024-08-30T19:14:32.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:14:32.616 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:03.342640+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:49:32.281029+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:01.967431+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:01:01.995704+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:08.045564+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:04:30.085414+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:04.955770+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:24:23.872343+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:09.054386+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:30:47.099929+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:14:01.993416+0000 37'5 37:62 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:14:43.023575+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:13.926003+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:52.974454+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:11.865499+0000 37'2 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:09:25.954653+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:07.032573+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:27:54.854230+0000 0 0 2024-08-30T19:14:32.617 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-30T19:14:05.603015+0000 37'6 37:56 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:06:29.959779+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:13:56.998645+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:56:06.197361+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:14:12.913711+0000 37'5 37:62 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:25:45.344467+0000 2 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:07.314048+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:52:58.809273+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:03.959608+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:46:06.626396+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:03.675344+0000 37'2 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:27:03.556910+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:06.310748+0000 37'2 37:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:45:37.412049+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:13:55.981254+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:32:44.020331+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:02.701866+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:04:04.419546+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:10.919311+0000 37'4 37:61 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:11:45.631205+0000 1 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:04.652704+0000 37'4 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:11:56.816167+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:02.371815+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:21:14.107565+0000 0 0 2024-08-30T19:14:32.618 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:08.519041+0000 37'3 37:52 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:54:41.116640+0000 1 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:05.297331+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:35:28.951861+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:13:57.955314+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:55:16.167653+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:06.553484+0000 37'3 37:53 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:03:43.507420+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:05.994071+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:53:48.977140+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:00.972852+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:53.256747+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:04.324535+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:53:41.733485+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:01.798221+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:31:32.512998+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:02.930167+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:08:46.610741+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:09.928288+0000 37'4 37:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:41:27.835716+0000 0 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:14:09.336154+0000 37'5 37:54 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:26:07.910586+0000 1 0 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:14:32.619 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:14:32.620 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:14:32.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:32.632 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.1 2024-08-30T19:14:32.632 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:14:32.632 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:14:32.633 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:14:32.633 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:14:32.633 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:14:32.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:14:32.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:14:32.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok config set objectstore_blackhole 1 2024-08-30T19:14:32.741 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:14:32.741 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:14:32.741 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:14:32.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:32.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-08-30T19:14:32.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:14:32.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:14:32.757 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:14:32.757 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:14:32.757 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:14:32.757 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:14:32.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:14:32.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-30T19:14:32.861 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:14:32.861 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:14:32.861 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:14:32.873 INFO:tasks.workunit.client.0.smithi079.stdout:create new divergent objects 2024-08-30T19:14:32.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-08-30T19:14:32.873 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-08-30T19:14:32.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:14:32.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-08-30T19:14:32.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:14:32.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-08-30T19:14:32.876 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent objects 2024-08-30T19:14:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-08-30T19:14:32.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-08-30T19:14:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-30T19:14:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-30T19:14:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-08-30T19:14:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-08-30T19:14:32.878 INFO:tasks.workunit.client.0.smithi079.stdout:remove divergent objects 2024-08-30T19:14:32.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-08-30T19:14:32.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-08-30T19:14:32.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-30T19:14:32.881 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-08-30T19:14:32.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-08-30T19:14:32.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-30T19:14:32.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-08-30T19:14:32.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-08-30T19:14:32.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-08-30T19:14:32.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-08-30T19:14:32.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-08-30T19:14:42.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-08-30T19:14:42.892 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-08-30T19:14:42.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-08-30T19:14:42.892 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-08-30T19:14:42.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:14:43.308 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:03.342640+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:49:32.281029+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:01.967431+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:01:01.995704+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:08.045564+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:04:30.085414+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:04.955770+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:24:23.872343+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:09.054386+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:30:47.099929+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 active+clean 2024-08-30T19:14:01.993416+0000 37'6 37:63 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:14:43.023575+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:13.926003+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:52.974454+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:11.865499+0000 37'2 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:09:25.954653+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:07.032573+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:27:54.854230+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-30T19:14:05.603015+0000 37'6 37:56 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:06:29.959779+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:13:56.998645+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:56:06.197361+0000 0 0 2024-08-30T19:14:43.309 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:14:12.913711+0000 37'5 37:62 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:25:45.344467+0000 2 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:07.314048+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:52:58.809273+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:03.959608+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:46:06.626396+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:03.675344+0000 37'2 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:27:03.556910+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:06.310748+0000 37'2 37:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:45:37.412049+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:13:55.981254+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:32:44.020331+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:02.701866+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:04:04.419546+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 active+clean 2024-08-30T19:14:10.919311+0000 37'6 37:63 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:11:45.631205+0000 1 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:04.652704+0000 37'4 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:11:56.816167+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:02.371815+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:21:14.107565+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 active+clean 2024-08-30T19:14:08.519041+0000 37'4 37:53 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:54:41.116640+0000 1 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:05.297331+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:35:28.951861+0000 0 0 2024-08-30T19:14:43.310 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:13:57.955314+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:55:16.167653+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:06.553484+0000 37'3 37:53 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:03:43.507420+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:05.994071+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:53:48.977140+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:00.972852+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:53.256747+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:14:04.324535+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:53:41.733485+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:14:01.798221+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:31:32.512998+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:14:02.930167+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:08:46.610741+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:14:09.928288+0000 37'4 37:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:41:27.835716+0000 0 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 active+clean 2024-08-30T19:14:09.336154+0000 37'7 37:56 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:26:07.910586+0000 1 0 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:14:43.311 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:14:43.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-08-30T19:14:43.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:14:43.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:14:43.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:14:43.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:14:43.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:14:43.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:14:43.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-30T19:14:43.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-08-30T19:14:44.080 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:14:44.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-30T19:14:44.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-08-30T19:14:44.667 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:14:44.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-30T19:14:44.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-08-30T19:14:45.268 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:14:45.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:45.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.1 2024-08-30T19:14:45.836 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already out. 2024-08-30T19:14:45.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:45.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-08-30T19:14:46.448 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-08-30T19:14:46.461 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 1 2 2024-08-30T19:14:46.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 1 2' 2024-08-30T19:14:46.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:03.342640+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:49:32.281029+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:01.967431+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:01:01.995704+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:08.045564+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:04:30.085414+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:04.955770+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:24:23.872343+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:09.054386+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:30:47.099929+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 stale+active+clean 2024-08-30T19:14:01.993416+0000 37'6 37:63 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:14:43.023575+0000 0 0 2024-08-30T19:14:46.892 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:13.926003+0000 37'3 37:60 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:52.974454+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-30T19:14:11.865499+0000 37'2 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:09:25.954653+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:07.032573+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:27:54.854230+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 stale+active+clean 2024-08-30T19:14:05.603015+0000 37'6 37:56 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:06:29.959779+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-30T19:13:56.998645+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:56:06.197361+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 stale+active+clean 2024-08-30T19:14:12.913711+0000 37'5 37:62 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:25:45.344467+0000 2 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-30T19:14:07.314048+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:52:58.809273+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:03.959608+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:46:06.626396+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-30T19:14:03.675344+0000 37'2 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:27:03.556910+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-30T19:14:06.310748+0000 37'2 37:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:45:37.412049+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:13:55.981254+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:32:44.020331+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:02.701866+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:04:04.419546+0000 0 0 2024-08-30T19:14:46.893 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 stale+active+clean 2024-08-30T19:14:10.919311+0000 37'6 37:63 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:11:45.631205+0000 1 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:04.652704+0000 37'4 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:11:56.816167+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:02.371815+0000 37'3 37:52 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:21:14.107565+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:08.519041+0000 37'4 37:53 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:54:41.116640+0000 1 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-30T19:14:05.297331+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:35:28.951861+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:13:57.955314+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:55:16.167653+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:06.553484+0000 37'3 37:53 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:03:43.507420+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:05.994071+0000 37'3 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:53:48.977140+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:00.972852+0000 37'4 37:57 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:53.256747+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-30T19:14:04.324535+0000 37'1 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:53:41.733485+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-30T19:14:01.798221+0000 37'3 37:53 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:31:32.512998+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-30T19:14:02.930167+0000 37'2 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:08:46.610741+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-30T19:14:09.928288+0000 37'4 37:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:41:27.835716+0000 0 0 2024-08-30T19:14:46.894 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 stale+active+clean 2024-08-30T19:14:09.336154+0000 37'7 37:56 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:26:07.910586+0000 1 0 2024-08-30T19:14:46.895 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:14:46.895 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:14:46.895 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:14:46.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:46.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-08-30T19:14:46.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:14:46.911 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:14:46.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:14:46.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:14:46.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:14:46.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:14:46.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:14:46.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:14:46.914 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:14:46.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:14:46.914 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:14:46.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:14:46.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:14:46.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:14:46.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:14:46.945 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:46.945+0000 7f01f51747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:46.947 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:46.949+0000 7f01f51747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:46.948 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:46.949+0000 7f01f51747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:47.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:14:47.337 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:14:47.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:47.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:14:47.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:48.173 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:48.173+0000 7f01f51747c0 -1 Falling back to public interface 2024-08-30T19:14:48.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:48.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:48.764 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:14:48.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:14:48.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:48.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:14:49.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:50.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:50.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:50.177 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:14:50.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:14:50.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:50.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:14:50.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:51.561 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:51.561+0000 7f01f51747c0 -1 osd.1 37 log_to_monitors true 2024-08-30T19:14:51.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:51.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:51.698 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:14:51.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:14:51.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:51.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:14:52.224 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up out weight 0 up_from 44 up_thru 35 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6802/3583746400,v1:127.0.0.1:6803/3583746400] [v2:127.0.0.1:6804/3583746400,v1:127.0.0.1:6805/3583746400] exists,up 9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:14:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:14:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:14:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:14:52.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:14:52.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:14:52.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:14:52.230 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:14:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:14:52.266 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:52.269+0000 7f52201867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:52.288 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:52.289+0000 7f52201867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:52.289 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:52.289+0000 7f52201867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:14:52.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:14:52.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:52.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:14:52.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:52.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:14:53.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:53.316 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:53.317+0000 7f52201867c0 -1 Falling back to public interface 2024-08-30T19:14:54.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:54.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:54.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:14:54.233 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:14:54.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:54.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:14:54.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:55.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:55.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:55.665 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:14:55.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:14:55.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:55.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:14:56.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:14:56.737 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:14:56.737+0000 7f52201867c0 -1 osd.2 37 log_to_monitors true 2024-08-30T19:14:57.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:14:57.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:14:57.133 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:14:57.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:14:57.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:14:57.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:14:57.603 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 46 up_thru 35 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/2765532185,v1:127.0.0.1:6811/2765532185] [v2:127.0.0.1:6812/2765532185,v1:127.0.0.1:6813/2765532185] exists,up e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:14:57.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:14:57.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:14:57.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:14:57.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:57.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.1 2024-08-30T19:14:58.213 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already in. 2024-08-30T19:14:58.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:14:58.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-08-30T19:14:58.893 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-08-30T19:14:58.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:14:58.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-08-30T19:14:58.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:14:58.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:14:58.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:14:58.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:14:58.913 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:14:58.913 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:14:58.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:14:58.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:14:58.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:14:58.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:14:58.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:14:58.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:14:58.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:14:58.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:14:58.974 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:59.523 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:14:59.739 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-08-30T19:14:59.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:14:59.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:14:59.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:14:59.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:59.741 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:14:59.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561028 2024-08-30T19:14:59.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561028 2024-08-30T19:14:59.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-188978561028' 2024-08-30T19:14:59.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:14:59.977 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495619 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495619 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-188978561028 2-197568495619' 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561028 2024-08-30T19:15:00.233 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:15:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:15:00.235 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561028 2024-08-30T19:15:00.235 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:15:00.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561028 2024-08-30T19:15:00.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-08-30T19:15:00.237 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 188978561028 2024-08-30T19:15:00.237 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:15:00.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561026 -lt 188978561028 2024-08-30T19:15:00.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:15:01.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:15:01.682 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:15:02.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561028 2024-08-30T19:15:02.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:15:02.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-197568495619 2024-08-30T19:15:02.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:15:02.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:15:02.160 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-197568495619 2024-08-30T19:15:02.160 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:15:02.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495619 2024-08-30T19:15:02.162 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 197568495619 2024-08-30T19:15:02.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-08-30T19:15:02.163 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:15:02.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495619 -lt 197568495619 2024-08-30T19:15:02.600 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:15:02.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:02.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:03.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-30T19:15:03.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:03.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:03.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:03.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:03.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:03.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:03.162 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:03.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:03.626 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:03.626 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:03.626 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:15:04.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:04.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:04.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:04.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:04.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:04.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:04.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:04.282 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:04.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:04.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:04.726 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:04.726 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:05.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:05.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:05.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:05.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:15:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:06.031 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:06.032 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:06.032 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:06.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:06.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:06.438 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:06.438 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:07.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:07.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:07.104 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:07.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:07.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:07.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:15:07.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:07.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:15:08.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:08.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:08.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:08.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:08.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:08.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:08.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:08.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:08.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:08.461 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:08.462 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:08.462 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:08.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:08.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:09.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:09.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:09.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:15:09.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:09.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:15:10.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:10.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:10.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:10.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:10.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:10.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:10.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:10.367 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:10.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:10.835 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:10.836 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:10.836 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:11.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:11.391 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:11.391 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:11.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:11.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:11.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:15:11.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:11.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:15:13.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:13.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:13.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:13.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:13.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:13.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:13.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:14.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:14.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:14.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:14.497 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:15.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:15.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:15.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:15:15.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:15.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:15:18.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:18.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:18.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:18.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:18.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:18.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:18.290 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:18.290 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:18.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:18.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:18.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:18.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:19.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:19.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:19.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:19.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:19.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:19.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:19.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:19.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:19.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:19.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:19.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:19.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:15:19.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:19.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:15:26.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:26.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:26.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:26.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:26.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:26.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:26.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:26.213 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:26.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:26.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:26.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:26.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:27.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:27.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:15:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:15:35.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:35.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:35.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:35.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:35.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:35.063 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:35.064 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:35.064 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:35.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:35.490 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:35.491 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:35.491 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:36.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:36.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:36.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:36.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:36.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:36.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:15:36.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "8392d738-c6f5-4abb-af75-10d8a0e7e482", 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:15:37.183002+0000", 2024-08-30T19:15:37.179 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:15:37.180 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.0 (root=default,host=smithi079) is down" 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.181 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 32 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.182 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.183 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.184 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.a is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.b is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.c is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.e is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.187 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [1,2]" 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,1]" 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.190 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:15:37.191 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:11:45.687106Z", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:11:45.687106Z", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:15:37.192 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:15:37.193 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.194 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:15:37.195 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 51, 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:11:46.591134+0000", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:14:59.497061+0000", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:14:57.062586+0000", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:14:58.692507+0000", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:15:37.196 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:15:37.197 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:13:44.569998+0000", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:15:37.198 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "on", 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 32, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 32, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 32, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 32, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 32, 2024-08-30T19:15:37.199 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "35", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "33", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-08-30T19:15:37.200 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:15:37.201 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:15:37.202 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.2400000095367432, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.2400000095367432, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-08-30T19:15:37.203 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 1.5, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 1.5, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "f556141b-5721-4aa9-9831-2ea62c079268", 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:15:37.204 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 5, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 35, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 38, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.205 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.206 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.207 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:15:37.208 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/1708408575", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/1708408575", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1708408575", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1708408575", 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:15:37.209 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9d444242-ce4b-4396-b69c-98884ed22ebf", 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 10, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 37, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 44, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 49, 2024-08-30T19:15:37.210 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 39, 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.212 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.213 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.214 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3583746400", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3583746400", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3583746400", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3583746400", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:15:37.215 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "e97e52bb-639a-403e-90ba-53ab43e5f83a", 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 37, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 46, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 49, 2024-08-30T19:15:37.216 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 40, 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.217 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.218 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.219 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:15:37.220 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/2765532185", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/2765532185", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2765532185", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2765532185", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.221 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:43.840464+0000", 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:12:23.905503+0000", 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:15:37.222 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:44.461061+0000", 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:13:01.938260+0000", 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:45.076332+0000", 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:15:37.223 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:13:40.597987+0000", 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:15:37.224 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:15:37.225 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/1708408575,v1:127.0.0.1:6805/1708408575]", 2024-08-30T19:15:37.226 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:15:37.227 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:11:53.722627Z", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:15:37.228 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/1708408575,v1:127.0.0.1:6803/1708408575]", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1708408575,v1:127.0.0.1:6809/1708408575]", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1708408575,v1:127.0.0.1:6807/1708408575]", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:15:37.229 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3583746400,v1:127.0.0.1:6805/3583746400]", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:15:37.230 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:15:37.231 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:12:31.365823Z", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:15:37.232 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3583746400,v1:127.0.0.1:6803/3583746400]", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3583746400,v1:127.0.0.1:6809/3583746400]", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3583746400,v1:127.0.0.1:6807/3583746400]", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:15:37.233 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/2765532185,v1:127.0.0.1:6813/2765532185]", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:15:37.234 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:15:37.235 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:13:10.055376Z", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/2765532185,v1:127.0.0.1:6811/2765532185]", 2024-08-30T19:15:37.236 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2765532185,v1:127.0.0.1:6817/2765532185]", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2765532185,v1:127.0.0.1:6815/2765532185]", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:15:37.237 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.238 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 44 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 46 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 51, 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.239 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:15:37.240 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:15:37.241 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.242 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:15:37.243 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:15:37.244 INFO:tasks.workunit.client.0.smithi079.stderr:report 2591672739 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:15:37.245 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:15:37.246 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.247 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.248 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.249 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.250 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.251 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:15:37.252 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:15:37.253 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:15:37.254 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:11:46:559121+0000", 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:15:37.255 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.256 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.257 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 32, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:15:37.258 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 95, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 95, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:15:37.259 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:15:37.260 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:15:37.261 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 100, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 100, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-08-30T19:15:37.262 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83872, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2328, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488928, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036662272, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2383872, 2024-08-30T19:15:37.263 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 543207, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10392, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416936 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:15:37.264 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:15:37.265 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83872, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2328, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488928, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036662272, 2024-08-30T19:15:37.266 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2383872, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 543207, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10392, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416936 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:15:37.267 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:15:37.268 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 61700, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:15:37.269 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 95, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 95, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:15:37.270 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:15:37.271 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:15:37.272 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 184566, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 100, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 100, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:15:37.273 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 21474836520 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 188978561035 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 197568495626 2024-08-30T19:15:37.274 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 32 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 8, 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-08-30T19:15:37.275 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 24, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.276 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:14:41.917473+0000", 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:15:37.277 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 104, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 437, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 593, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:15:37.278 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-08-30T19:15:37.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:15:37.603 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:15:37.603 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 51:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 51:70 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 51:64 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:15:37.604 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 51:73 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 51:62 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 37'4 51:72 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 51:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 51:63 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:15:37.605 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 51:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 51:65 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:15:37.606 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:15:37.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-08-30T19:15:37.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:15:37.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-08-30T19:15:37.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:15:37.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:15:37.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:15:37.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:15:37.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:15:38.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:15:38.244 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:15:38.244 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:15:38.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:15:38.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:15:38.244 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:15:38.381 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-08-30T19:15:38.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:15:38.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:15:38.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:15:38.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:15:38.382 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:15:38.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561038 2024-08-30T19:15:38.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561038 2024-08-30T19:15:38.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-188978561038' 2024-08-30T19:15:38.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:15:38.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:15:38.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495629 2024-08-30T19:15:38.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495629 2024-08-30T19:15:38.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-188978561038 2-197568495629' 2024-08-30T19:15:38.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:15:38.718 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561038 2024-08-30T19:15:38.718 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:15:38.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:15:38.719 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561038 2024-08-30T19:15:38.719 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:15:38.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561038 2024-08-30T19:15:38.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561038' 2024-08-30T19:15:38.721 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 188978561038 2024-08-30T19:15:38.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:15:39.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561036 -lt 188978561038 2024-08-30T19:15:39.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:15:40.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:15:40.129 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:15:40.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561038 -lt 188978561038 2024-08-30T19:15:40.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:15:40.542 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-197568495629 2024-08-30T19:15:40.542 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:15:40.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:15:40.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-197568495629 2024-08-30T19:15:40.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:15:40.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495629 2024-08-30T19:15:40.546 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 197568495629 2024-08-30T19:15:40.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 197568495629' 2024-08-30T19:15:40.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:15:40.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495629 -lt 197568495629 2024-08-30T19:15:40.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:15:40.970 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:40.970 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:41.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-30T19:15:41.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:41.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:41.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:41.992 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:41.992 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:41.992 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:42.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:42.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:15:42.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:15:42.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:15:42.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:42.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:15:42.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:42.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:42.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:42.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:42.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:42.618 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:42.618 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:42.618 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:43.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:43.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:43.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:43.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:43.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:43.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:43.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:43.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:43.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:43.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:43.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:43.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:43.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:44.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:44.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:44.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:15:44.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:44.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:15:44.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:44.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:44.320 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:44.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:44.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:44.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:44.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:44.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:44.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:44.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:44.759 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:44.759 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:45.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:45.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:45.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:45.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:45.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:45.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:15:45.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:45.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:15:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:46.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:46.282 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:46.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:46.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:46.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:46.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:46.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:46.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:46.716 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:46.717 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:46.717 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:47.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:47.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:47.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:47.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:47.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:47.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:15:47.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:47.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:15:48.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:48.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:48.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:48.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:48.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:48.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:48.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:48.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:49.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:49.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:49.084 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:49.084 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:49.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:49.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:49.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:49.614 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:50.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:50.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:50.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:15:50.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:50.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:15:51.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:51.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:51.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:51.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:51.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:51.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:51.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:51.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:52.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:52.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:52.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:52.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:52.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:52.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:53.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:53.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:53.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:15:53.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:53.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:15:56.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:15:56.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:15:56.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:15:56.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:15:56.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:15:56.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:15:56.582 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:15:56.582 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:15:57.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:15:57.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:15:57.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:15:57.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:15:57.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:15:57.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:15:57.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:15:57.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:15:57.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:15:57.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:15:57.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:15:57.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:15:57.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:15:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:15:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:15:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:15:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:15:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:16:04.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:16:04.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:16:04.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:16:04.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:16:04.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:16:04.533 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:16:04.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:16:04.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:16:04.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:16:04.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:16:04.962 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:16:04.962 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:16:05.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:16:05.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:16:05.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:16:05.514 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:16:06.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:16:06.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:16:06.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:16:06.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:16:06.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:16:13.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:16:13.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:16:13.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:16:13.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:16:13.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:16:13.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:16:13.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:16:13.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:16:13.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:16:13.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:16:13.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:16:13.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:16:14.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:16:14.333 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:16:14.334 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:16:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:16:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:16:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:16:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "8392d738-c6f5-4abb-af75-10d8a0e7e482", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:16:15.483149+0000", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:16:15.480 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:16:15.481 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.0 (root=default,host=smithi079) is down" 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 64 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:16:15.482 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.2 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.3 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.483 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.4 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.5 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.6 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.7 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.484 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.8 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.9 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.a is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.b is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.c is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.485 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.e is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.486 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.11 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.12 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.13 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.15 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.487 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.16 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.17 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.18 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.19 is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1a is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1b is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1c is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1d is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.489 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1e is stuck undersized for 74s, current state active+undersized+degraded, last acting [1,2]" 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:16:15.490 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:16:15.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:11:45.687106Z", 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:11:45.687106Z", 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:16:15.492 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:16:15.493 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:16:15.494 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:16:15.495 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 51, 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:11:46.591134+0000", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:14:59.497061+0000", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:14:57.062586+0000", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:14:58.692507+0000", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:16:15.496 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:16:15.497 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:13:44.569998+0000", 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-08-30T19:16:15.498 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "on", 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 32, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 32, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 32, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 32, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 32, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:16:15.499 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "35", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "33", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-08-30T19:16:15.500 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:16:15.501 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:16:15.502 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.2400000095367432, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.2400000095367432, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 1.5, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 1.5, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-08-30T19:16:15.503 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "f556141b-5721-4aa9-9831-2ea62c079268", 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:16:15.504 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 5, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 35, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 38, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.505 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:16:15.506 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:16:15.507 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.508 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1708408575 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/1708408575", 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/1708408575", 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1708408575", 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1708408575", 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.509 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9d444242-ce4b-4396-b69c-98884ed22ebf", 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 10, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 37, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 44, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 49, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 39, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:16:15.510 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:16:15.511 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:16:15.512 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.513 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3583746400 2024-08-30T19:16:15.514 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3583746400", 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3583746400", 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3583746400", 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3583746400", 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "e97e52bb-639a-403e-90ba-53ab43e5f83a", 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 37, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 46, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 49, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 40, 2024-08-30T19:16:15.516 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.517 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.518 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.519 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2765532185 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.520 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/2765532185", 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/2765532185", 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2765532185", 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2765532185", 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.521 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:43.840464+0000", 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:12:23.905503+0000", 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:44.461061+0000", 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:16:15.522 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:13:01.938260+0000", 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:14:45.076332+0000", 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:13:40.597987+0000", 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:16:15.523 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:16:15.524 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:16:15.525 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/1708408575,v1:127.0.0.1:6805/1708408575]", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:16:15.526 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.527 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:11:53.722627Z", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/1708408575,v1:127.0.0.1:6803/1708408575]", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1708408575,v1:127.0.0.1:6809/1708408575]", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1708408575,v1:127.0.0.1:6807/1708408575]", 2024-08-30T19:16:15.528 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:16:15.529 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3583746400,v1:127.0.0.1:6805/3583746400]", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:16:15.530 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:16:15.531 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:12:31.365823Z", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3583746400,v1:127.0.0.1:6803/3583746400]", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3583746400,v1:127.0.0.1:6809/3583746400]", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3583746400,v1:127.0.0.1:6807/3583746400]", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:16:15.532 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.533 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/2765532185,v1:127.0.0.1:6813/2765532185]", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:16:15.534 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:13:10.055376Z", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:16:15.535 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/2765532185,v1:127.0.0.1:6811/2765532185]", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2765532185,v1:127.0.0.1:6817/2765532185]", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2765532185,v1:127.0.0.1:6815/2765532185]", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:16:15.536 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:16:15.537 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 44 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:16:15.538 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 46 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 51, 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:16:15.539 INFO:tasks.workunit.client.0.smithi079.stderr:report 4196041489 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:16:15.540 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.541 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.542 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:16:15.543 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:16:15.544 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.545 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:16:15.546 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.547 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.548 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:16:15.549 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.550 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.551 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:16:15.552 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:16:15.553 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:16:15.554 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:11:46:559121+0000", 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:16:15.555 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:16:15.556 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-08-30T19:16:15.557 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 32, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62050, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:16:15.558 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 96, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 96, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:16:15.559 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:16:15.560 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:16:15.561 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 101, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 101, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.562 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83872, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2328, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488928, 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:16:15.563 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036662272, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2383872, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 543907, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10392, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416936 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:16:15.564 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:16:15.565 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83872, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2328, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:16:15.566 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488928, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036662272, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2383872, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 543907, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10392, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416936 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:16:15.567 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:16:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62050, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-08-30T19:16:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 96, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 96, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:16:15.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:16:15.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-08-30T19:16:15.572 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 185266, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 101, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 101, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.573 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 21474836520 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 188978561045 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 197568495636 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 32 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 8, 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.575 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 24, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:16:15.576 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 7, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:15:59.944678+0000", 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 123, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 504, 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:16:15.577 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 679, 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:16:15.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-08-30T19:16:15.579 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-08-30T19:16:15.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:15.885 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:15.885 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 51:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 51:70 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 51:64 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 51:73 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:15.886 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 51:62 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 51:73 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 51:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 51:63 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 51:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 51:65 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:15.888 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:15.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:16:15.901 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.1 2024-08-30T19:16:15.901 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:16:15.901 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:16:15.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok set_recovery_delay 100000 2024-08-30T19:16:16.008 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:16:16.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-08-30T19:16:16.021 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-08-30T19:16:16.021 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:16:16.021 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:16:16.021 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:16.022 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:16.022 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:16.022 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:16:16.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:16:16.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok set_recovery_delay 100000 2024-08-30T19:16:16.120 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:16:16.131 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 0 2024-08-30T19:16:16.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 0' 2024-08-30T19:16:16.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:16.514 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 51:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 51:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 51:70 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 51:64 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 51:73 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 51:62 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:16.515 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 51:73 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 51:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 51:60 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 51:63 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 51:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 51:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 51:61 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 51:61 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:16.516 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 51:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:16.517 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 51:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:16.517 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 51:65 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:16.517 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:16.517 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:16.517 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:16.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-08-30T19:16:17.310 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:16:17.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:16:17.335 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:16:17.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:16:17.338 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:16:17.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:16:17.339 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:16:17.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:16:17.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:16:17.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:16:17.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:16:17.379 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:17.384+0000 7fec7f37a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:17.382 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:17.388+0000 7fec7f37a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:17.383 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:17.388+0000 7fec7f37a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:17.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-30T19:16:17.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-08-30T19:16:18.649 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:18.652+0000 7fec7f37a7c0 -1 Falling back to public interface 2024-08-30T19:16:21.786 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:21.787+0000 7fec7f37a7c0 -1 osd.0 37 log_to_monitors true 2024-08-30T19:16:22.758 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-08-30T19:16:22.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-08-30T19:16:22.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 53:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 53:75 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:23.182 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 53:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 53:75 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 53:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 53:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 52:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 53:70 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 53:77 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 53:65 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 53:66 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:23.183 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 53:77 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 52:64 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 53:67 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 53:65 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 53:71 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 53:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 53:65 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 53:70 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:23.184 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 53:76 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:23.185 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 53:69 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:23.185 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:23.185 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:23.185 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:23.199 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.0 2024-08-30T19:16:23.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:16:23.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:16:23.201 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:23.201 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:23.201 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:23.201 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:16:23.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:16:23.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok set_recovery_delay 100000 2024-08-30T19:16:23.315 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:16:23.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-08-30T19:16:24.191 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:16:24.215 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:24.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:16:24.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:16:24.868 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3558038836,v1:127.0.0.1:6819/3558038836] [v2:127.0.0.1:6820/3558038836,v1:127.0.0.1:6821/3558038836] exists,up f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:16:24.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:16:24.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:16:24.869 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:24.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:16:24.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:16:25.515 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 44 up_thru 56 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6802/3583746400,v1:127.0.0.1:6803/3583746400] [v2:127.0.0.1:6804/3583746400,v1:127.0.0.1:6805/3583746400] exists,up 9d444242-ce4b-4396-b69c-98884ed22ebf 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:16:25.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:16:25.517 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:25.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:16:25.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:16:25.956 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 46 up_thru 56 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/2765532185,v1:127.0.0.1:6811/2765532185] [v2:127.0.0.1:6812/2765532185,v1:127.0.0.1:6813/2765532185] exists,up e97e52bb-639a-403e-90ba-53ab43e5f83a 2024-08-30T19:16:25.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:16:25.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:16:25.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:16:25.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:26.366 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:26.366 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:26.366 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:26.366 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:26.366 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 54:79 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 54:76 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 54:70 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 54:79 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:26.367 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 54:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 54:79 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 54:68 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 54:69 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:26.368 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 54:78 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 54:71 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:26.369 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:26.382 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-08-30T19:16:26.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-08-30T19:16:26.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:26.798 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 54:79 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 54:76 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 54:70 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 54:79 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 54:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:26.799 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 54:79 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 54:68 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 54:69 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:26.800 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 54:78 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:26.801 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 54:71 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:26.801 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:26.801 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:26.801 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:26.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-08-30T19:16:26.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 0' 2024-08-30T19:16:26.971 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 0 2024-08-30T19:16:26.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:27.358 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:27.358 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.195025+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:40:40.163637+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532660+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:06:22.496378+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.533432+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:58:00.613272+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535421+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T03:23:31.370631+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533665+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:03:54.404898+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.536661+0000 37'5 54:79 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:56:31.260924+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533986+0000 37'3 54:77 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:00:26.901519+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534219+0000 37'2 54:76 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:48:46.917238+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532447+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:05:48.456106+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-30T19:15:00.167728+0000 37'6 54:70 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:43:19.028231+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534525+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:55:22.410178+0000 0 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531403+0000 37'5 54:79 [1,2] 1 [1,2] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:58:05.064713+0000 2 0 2024-08-30T19:16:27.359 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.531535+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:29:30.790431+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.533483+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:42.527351+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:15:00.193655+0000 37'2 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:11:26.639478+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.536292+0000 37'2 54:68 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:47:19.862920+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531279+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:14:50.519493+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193752+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:43:32.220041+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.531509+0000 51'5 54:79 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:29:53.016076+0000 1 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:15:00.193644+0000 37'4 54:68 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:14:20.801837+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194496+0000 37'3 54:66 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:56:21.972885+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.532334+0000 37'3 54:69 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:30:42.487294+0000 1 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.537381+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:09.820900+0000 0 0 2024-08-30T19:16:27.360 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.535670+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:14.955994+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.194832+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T03:37:24.129793+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:14:59.531524+0000 37'3 54:73 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:57:18.487158+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.531041+0000 37'4 54:74 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:51:42.440872+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-30T19:14:59.535193+0000 37'1 54:67 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T19:16:58.525142+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-30T19:15:00.193636+0000 37'3 54:67 [2,1] 2 [2,1] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:42:46.825478+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-30T19:14:59.534228+0000 37'2 54:72 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:19:55.976089+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-30T19:14:59.532852+0000 37'4 54:78 [1,2] 1 [1,2] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T02:03:51.003849+0000 0 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-30T19:14:59.532671+0000 37'5 54:71 [1,2] 1 [1,2] 1 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T21:33:20.618344+0000 1 0 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:27.361 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:27.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.0 2024-08-30T19:16:27.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:16:27.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:16:27.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:16:27.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:16:27.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27261.out 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27261.out 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-30T19:16:27.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27261.out 2024-08-30T19:16:29.224 INFO:tasks.workunit.client.0.smithi079.stderr:Exporting 1.1b info 1.1b( v 37'5 lc 0'0 (0'0,37'5] local-lis/les=56/57 n=5 ec=33/23 lis/c=56/35 les/c/f=57/37/0 sis=56) 2024-08-30T19:16:29.225 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-08-30T19:16:29.231 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-08-30T19:16:29.232 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd677bd3:::existing_35:head# 2024-08-30T19:16:29.239 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-08-30T19:16:29.240 INFO:tasks.workunit.client.0.smithi079.stderr:Export successful 2024-08-30T19:16:29.241 INFO:tasks.workunit.client.0.smithi079.stdout: marking collection for removal 2024-08-30T19:16:29.241 INFO:tasks.workunit.client.0.smithi079.stdout:setting '_remove' omap key 2024-08-30T19:16:29.241 INFO:tasks.workunit.client.0.smithi079.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-08-30T19:16:29.340 INFO:tasks.workunit.client.0.smithi079.stderr:Remove successful 2024-08-30T19:16:29.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op import --file td/divergent-priors/exp.27261.out 2024-08-30T19:16:29.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:16:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:16:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-30T19:16:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:16:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-30T19:16:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op import --file td/divergent-priors/exp.27261.out 2024-08-30T19:16:31.147 INFO:tasks.workunit.client.0.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e58 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-08-30T19:16:31.147 INFO:tasks.workunit.client.0.smithi079.stdout:Importing pgid 1.1b 2024-08-30T19:16:31.148 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-08-30T19:16:31.148 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:16:31.226 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-08-30T19:16:31.227 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:16:31.268 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd677bd3:::existing_35:head# 2024-08-30T19:16:31.268 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:16:31.310 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-08-30T19:16:31.310 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-08-30T19:16:31.352 INFO:tasks.workunit.client.0.smithi079.stdout:write_pg epoch 57 info 1.1b( v 37'5 lc 0'0 (0'0,37'5] local-lis/les=56/57 n=5 ec=33/23 lis/c=56/35 les/c/f=57/37/0 sis=56) 2024-08-30T19:16:31.393 INFO:tasks.workunit.client.0.smithi079.stdout:Import successful 2024-08-30T19:16:31.932 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 0 2024-08-30T19:16:31.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 0' 2024-08-30T19:16:31.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:32.350 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:32.350 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.186598+0000 37'3 58:66 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:12:58.443021+0000 0 0 2024-08-30T19:16:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.191730+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:43:57.199308+0000 0 0 2024-08-30T19:16:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:25.201100+0000 37'4 58:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:38:58.501132+0000 0 0 2024-08-30T19:16:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.193338+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T01:17:13.739953+0000 0 0 2024-08-30T19:16:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.190951+0000 37'3 58:91 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:21:57.915229+0000 0 0 2024-08-30T19:16:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:25.199463+0000 37'5 58:92 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.188357+0000 37'3 58:91 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:01:37.851938+0000 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:25.190854+0000 37'2 58:90 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:46:18.785314+0000 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:25.195815+0000 58'5 58:89 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:31:08.188451+0000 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-30T19:16:25.710591+0000 37'6 58:83 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:25:54.914012+0000 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:25.200485+0000 37'2 58:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:41:05.962426+0000 0 0 2024-08-30T19:16:32.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:25.188811+0000 37'5 58:93 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:00:06.857949+0000 2 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:25.183660+0000 37'1 58:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:25:08.452301+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.193793+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T07:11:15.403413+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:25.613497+0000 37'2 58:79 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:38:18.858711+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:25.198207+0000 37'2 58:65 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:29:57.845811+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.192318+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:24:39.042978+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.707568+0000 37'3 58:80 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:02:34.436432+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:25.201565+0000 51'5 58:92 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:25.710066+0000 37'4 58:81 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:45:25.749551+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.185872+0000 37'3 58:66 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T04:59:41.201353+0000 0 0 2024-08-30T19:16:32.353 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-08-30T19:16:25.183443+0000 37'3 58:65 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:25.189886+0000 37'1 58:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:55:11.181143+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.199917+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:39:02.361866+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.709451+0000 37'3 58:80 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:20:41.338001+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.196686+0000 37'3 58:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T02:36:56.293976+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:25.194414+0000 37'4 58:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:49:01.689794+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:25.198995+0000 37'1 58:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:46:48.947480+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:25.613528+0000 37'3 58:80 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:18:46.074189+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:25.185086+0000 37'2 58:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:23:34.790122+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:25.194720+0000 37'4 58:92 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T07:04:23.482632+0000 0 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovering+degraded 2024-08-30T19:16:25.197189+0000 37'5 58:68 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:32.354 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:32.355 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:16:32.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:16:32.368 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:16:32.369 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:32.369 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:16:32.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:16:32.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:16:32.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:16:32.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:16:32.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:16:32.371 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:16:32.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:16:32.410 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:32.415+0000 7f538cbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:32.412 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:32.415+0000 7f538cbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:32.414 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:32.419+0000 7f538cbf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:32.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:16:32.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:16:32.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:16:32.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:16:32.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:16:33.333 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3558038836,v1:127.0.0.1:6819/3558038836] [v2:127.0.0.1:6820/3558038836,v1:127.0.0.1:6821/3558038836] exists,up f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:16:33.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:16:33.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:16:33.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 0 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:16:33.335 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:33.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:16:33.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:16:33.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:16:33.379 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:33.383+0000 7f538cbf97c0 -1 Falling back to public interface 2024-08-30T19:16:33.809 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3558038836,v1:127.0.0.1:6819/3558038836] [v2:127.0.0.1:6820/3558038836,v1:127.0.0.1:6821/3558038836] exists,up f556141b-5721-4aa9-9831-2ea62c079268 2024-08-30T19:16:33.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:16:33.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:16:33.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:16:33.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-08-30T19:16:36.400 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:36.403+0000 7f538cbf97c0 -1 osd.0 58 log_to_monitors true 2024-08-30T19:16:53.812 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.0 2024-08-30T19:16:53.813 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:16:53.813 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:16:53.813 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:53.813 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:53.813 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:53.814 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:16:53.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-08-30T19:16:53.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok dump_ops_in_flight 2024-08-30T19:16:53.930 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:16:53.930 INFO:tasks.workunit.client.0.smithi079.stdout: "ops": [], 2024-08-30T19:16:53.930 INFO:tasks.workunit.client.0.smithi079.stdout: "num_ops": 0 2024-08-30T19:16:53.930 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:16:53.942 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-08-30T19:16:53.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-08-30T19:16:53.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.579624+0000 37'3 63:76 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:54:23.670329+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.580939+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:04:14.509089+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.579201+0000 37'4 63:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:18:04.982543+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.585291+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T00:41:33.621868+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.581582+0000 37'3 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:24:18.562275+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:38.584458+0000 37'5 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.578573+0000 37'3 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:23:13.763559+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.583635+0000 37'2 63:103 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:11:45.299386+0000 0 0 2024-08-30T19:16:54.336 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:38.580371+0000 58'5 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T07:13:06.087403+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-30T19:16:38.577772+0000 37'6 63:97 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:35:44.733377+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.579845+0000 37'2 63:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:20:35.148291+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:38.572198+0000 37'5 63:106 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:06:31.279313+0000 2 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.576202+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:09:19.717507+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.582885+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:13:48.361439+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.584872+0000 37'2 63:93 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:59:12.831492+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.576598+0000 37'2 63:75 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:29:25.952844+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.572185+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:17:38.035843+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.572695+0000 37'3 63:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:35:45.027116+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-08-30T19:16:38.589166+0000 51'5 63:105 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.577154+0000 37'4 63:95 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:33:33.440102+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.580959+0000 37'3 63:76 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:26:21.036037+0000 0 0 2024-08-30T19:16:54.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-08-30T19:16:38.576867+0000 37'3 63:74 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.578040+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:14:31.479713+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.584031+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:03:19.487283+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.426978+0000 37'3 63:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:54:53.152674+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.579187+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:10:28.772438+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.574638+0000 37'4 63:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:57:30.843983+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.574125+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:35:03.054990+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.574134+0000 37'3 63:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:14:20.963003+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.582225+0000 37'2 63:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:17:30.517587+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.584691+0000 37'4 63:105 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T20:21:43.191103+0000 0 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:38.577931+0000 37'5 63:76 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:54.338 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:54.339 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:54.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-30T19:16:54.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-30T19:16:54.499 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:16:54.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-30T19:16:54.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-30T19:16:54.666 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:16:54.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-30T19:16:54.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-30T19:16:54.825 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:16:54.839 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-08-30T19:16:54.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-08-30T19:16:54.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.579624+0000 37'3 63:76 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:03.342545+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:54:23.670329+0000 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.580939+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:01.967314+0000 0'0 2024-08-30T19:14:01.967314+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:04:14.509089+0000 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.579201+0000 37'4 63:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:08.045390+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:18:04.982543+0000 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.585291+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:04.955692+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T00:41:33.621868+0000 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.581582+0000 37'3 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.054210+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:24:18.562275+0000 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:38.584458+0000 37'5 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:13:59.955067+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 0 0 2024-08-30T19:16:55.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.578573+0000 37'3 63:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:13.925821+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:23:13.763559+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.583635+0000 37'2 63:103 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:11.865204+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:11:45.299386+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:38.580371+0000 58'5 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:07.032456+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T07:13:06.087403+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-30T19:16:38.577772+0000 37'6 63:97 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:05.602844+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T00:35:44.733377+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.579845+0000 37'2 63:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:56.998480+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:20:35.148291+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-30T19:16:38.572198+0000 37'5 63:106 [1,2,0] 1 [1,2,0] 1 37'2 2024-08-30T19:14:12.913517+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:06:31.279313+0000 2 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.576202+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:07.313946+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:09:19.717507+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.582885+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:03.959449+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:13:48.361439+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.584872+0000 37'2 63:93 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:03.675160+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:59:12.831492+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.576598+0000 37'2 63:75 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:06.310653+0000 0'0 2024-08-30T19:14:06.310653+0000 0 0 periodic scrub scheduled @ 2024-08-31T22:29:25.952844+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.572185+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:55.981133+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T01:17:38.035843+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.572695+0000 37'3 63:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:02.701738+0000 0'0 2024-08-30T19:14:02.701738+0000 0 1 periodic scrub scheduled @ 2024-08-31T23:35:45.027116+0000 0 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-08-30T19:16:38.589166+0000 51'5 63:105 [1,2,0] 1 [1,2,0] 1 37'1 2024-08-30T19:14:10.919184+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:55.222 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.577154+0000 37'4 63:95 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:04.652401+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T04:33:33.440102+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.580959+0000 37'3 63:76 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-30T19:14:02.371734+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:26:21.036037+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-08-30T19:16:38.576867+0000 37'3 63:74 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:08.518838+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.578040+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:05.297046+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T20:14:31.479713+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.584031+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:13:57.955170+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T05:03:19.487283+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.426978+0000 37'3 63:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-30T19:14:06.553298+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:54:53.152674+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.579187+0000 37'3 63:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:05.993987+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T22:10:28.772438+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.574638+0000 37'4 63:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:00.972590+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:57:30.843983+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-30T19:16:38.574125+0000 37'1 63:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-30T19:14:04.324336+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-09-01T05:35:03.054990+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-30T19:16:38.574134+0000 37'3 63:94 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-30T19:14:01.798151+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-08-31T19:14:20.963003+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-30T19:16:38.582225+0000 37'2 63:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:14:02.929906+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 periodic scrub scheduled @ 2024-09-01T06:17:30.517587+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-30T19:16:38.584691+0000 37'4 63:105 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-30T19:14:09.928114+0000 0'0 2024-08-30T19:13:44.640725+0000 0 0 periodic scrub scheduled @ 2024-08-31T20:21:43.191103+0000 0 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-30T19:16:38.577931+0000 37'5 63:76 [0,1,2] 0 [0,1,2] 0 37'1 2024-08-30T19:14:09.335998+0000 0'0 2024-08-30T19:13:44.640725+0000 0 1 no scrub is scheduled 1 0 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:16:55.223 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:16:55.224 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:16:55.237 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-08-30T19:16:55.238 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-08-30T19:16:55.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:16:55.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-08-30T19:16:55.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:16:55.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-08-30T19:16:55.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:16:55.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-08-30T19:16:55.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:16:55.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-08-30T19:16:55.378 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-08-30T19:16:55.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:16:55.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-08-30T19:16:55.411 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_1: (2) No such file or directory 2024-08-30T19:16:55.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-30T19:16:55.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-08-30T19:16:55.445 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_2: (2) No such file or directory 2024-08-30T19:16:55.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-08-30T19:16:55.448 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-30T19:16:55.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:16:55.465 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4676.0:1 2024-08-30T19:14:32.946883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.465 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4676.0:1 2024-08-30T19:14:32.946883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.465 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head has no more recent entries in log 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head olog_can_rollback_to: 37'6 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot rollback 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4676.0:1 2024-08-30T19:14:32.946883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot roll back, removing and adding to missing 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4669.0:1 2024-08-30T19:14:32.940368+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 20 _merge_object_divergent_entries: keeping 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4669.0:1 2024-08-30T19:14:32.940368+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'5 last_divergent_update: 37'5 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.463+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-30T19:16:55.466 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'6 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.472 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:16:24.539+0000 7f01d70fa640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4677.0:1 2024-08-30T19:14:32.949302+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: keeping 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4677.0:1 2024-08-30T19:14:32.949302+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 37'1 first_divergent_update: 37'4 last_divergent_update: 37'4 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'4 2024-08-30T19:16:55.473 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4677.0:1 2024-08-30T19:14:32.949302+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4674.0:1 2024-08-30T19:14:32.945592+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: keeping 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4674.0:1 2024-08-30T19:14:32.945592+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 37'1 first_divergent_update: 37'7 last_divergent_update: 37'7 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 37'7 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4674.0:1 2024-08-30T19:14:32.945592+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 37'6 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4662.0:1 2024-08-30T19:14:32.917655+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 20 _merge_object_divergent_entries: keeping 37'6 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4662.0:1 2024-08-30T19:14:32.917655+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.474 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.539+0000 7fec62302640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-08-30T19:16:55.475 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'6 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4665.0:1 2024-08-30T19:14:32.928160+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4676.0:1 2024-08-30T19:14:32.946883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4676.0:1 2024-08-30T19:14:32.946883+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: more recent entry found: 51'5 (37'1) modify 1:70b3bf12:::existing_4:head by client.4794.0:1 2024-08-30T19:15:37.664370+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4669.0:1 2024-08-30T19:14:32.940368+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 20 _merge_object_divergent_entries: keeping 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4669.0:1 2024-08-30T19:14:32.940368+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:16:55.476 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'5 last_divergent_update: 37'5 2024-08-30T19:16:55.477 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-30T19:16:55.477 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-08-30T19:16:24.939+0000 7fec61300640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-30T19:16:55.477 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-30T19:16:55.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:16:55.497 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-30T19:16:55.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:16:55.512 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-08-30T19:16:55.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-08-30T19:16:55.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27261.out 2024-08-30T19:16:55.514 INFO:tasks.workunit.client.0.smithi079.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27261.out': No such file or directory 2024-08-30T19:16:55.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-08-30T19:16:55.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-30T19:16:55.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-30T19:16:56.006 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-08-30T19:16:56.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-08-30T19:16:56.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:16:56.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:16:56.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:16:56.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:16:56.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:16:57.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:16:57.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-30T19:16:57.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:16:57.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:16:57.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:16:57.338 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:16:57.338 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:16:57.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:16:57.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:16:57.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:16:57.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:16:57.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:16:57.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:16:57.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:16:57.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:16:57.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:16:57.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:16:57.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:16:57.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:16:57.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:16:57.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:16:57.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:16:57.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:16:57.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:16:57.388 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:16:57.388 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:57.388 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:57.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:16:57.390 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:16:57.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:16:57.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:16:57.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:16:57.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:16:57.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:16:57.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:16:57.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:16:57.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:16:57.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:16:57.394 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:16:57.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:16:57.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:16:57.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:16:57.396 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:16:57.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:16:57.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:16:57.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:16:57.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:16:57.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:57.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:57.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:16:57.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:16:57.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:16:57.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-30T19:16:57.401 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:16:57.401 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:57.401 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:57.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27261 2024-08-30T19:16:57.403 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:16:57.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-30T19:16:57.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-08-30T19:16:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-08-30T19:16:57.405 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-30T19:16:57.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-30T19:16:57.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:16:57.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:16:57.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:16:57.803 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:16:57.803 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:57.803 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:57.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:57.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:16:58.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:16:58.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:16:58.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:16:58.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:16:58.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:16:58.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:58.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:16:58.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:16:58.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get fsid 2024-08-30T19:16:58.141 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:16:58.141 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:16:58.141 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:16:58.141 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:16:58.142 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:58.143 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:58.143 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-mon.a.asok 2024-08-30T19:16:58.143 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:16:58.143 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27261/ceph-mon.a.asok config get mon_host 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-30T19:16:58.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:59.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:59.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:16:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:16:59.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:16:59.257 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:16:59.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:16:59.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:16:59.259 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:16:59.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:16:59.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3f69e6db-716e-4819-87c9-5d2444ea98b7' 2024-08-30T19:16:59.260 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:16:59.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:16:59.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrGtJmcAUvERAAXxpDFOiuYGXCrKMlX/5cgQ== 2024-08-30T19:16:59.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrGtJmcAUvERAAXxpDFOiuYGXCrKMlX/5cgQ=="}' 2024-08-30T19:16:59.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3f69e6db-716e-4819-87c9-5d2444ea98b7 -i td/divergent-priors/0/new.json 2024-08-30T19:16:59.608 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:16:59.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-30T19:16:59.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrGtJmcAUvERAAXxpDFOiuYGXCrKMlX/5cgQ== --osd-uuid 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:16:59.666 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:59.670+0000 7f62c53c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:59.686 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:59.690+0000 7f62c53c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:59.687 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:59.690+0000 7f62c53c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:16:59.688 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:59.690+0000 7f62c53c37c0 -1 bdev(0x558455078a80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:16:59.688 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:16:59.690+0000 7f62c53c37c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-30T19:17:06.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-30T19:17:06.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:17:06.219 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:17:06.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:17:06.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:17:06.890 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:17:06.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:17:06.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:17:06.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:17:06.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:17:06.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:17:06.933 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:06.934+0000 7ff80842a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:06.958 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:06.962+0000 7ff80842a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:06.959 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:06.962+0000 7ff80842a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:07.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:17:07.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:17:07.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:17:07.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:17:07.324 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:17:07.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:17:07.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:07.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:17:07.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:07.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:07.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:08.045 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:08.046+0000 7ff80842a7c0 -1 Falling back to public interface 2024-08-30T19:17:08.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:08.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:08.759 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:17:08.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:17:08.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:08.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:09.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:10.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:10.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:10.188 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:17:10.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:17:10.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:10.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:10.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:10.961 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:10.962+0000 7ff80842a7c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:17:11.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:11.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:11.622 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:17:11.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:17:11.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:11.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:12.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:13.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:13.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:13.172 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:17:13.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:17:13.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:13.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:13.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:14.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:14.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:14.635 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:17:14.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:17:14.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:14.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:15.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:16.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:16.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:16.236 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:17:16.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:17:16.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:16.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:16.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:17.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:17.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:17.673 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:17:17.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:17:17.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:17.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:18.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:19.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:19.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:19.236 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:17:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:17:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:19.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:20.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:20.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:20.695 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:17:20.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:17:20.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:20.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:21.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:22.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:22.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:22.201 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:17:22.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:17:22.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:22.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:22.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:23.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:23.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:23.694 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:17:23.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:17:23.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:23.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:24.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:25.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:25.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:25.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:17:25.421 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:17:25.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:25.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:25.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:26.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:26.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:26.862 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:17:26.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:17:26.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:26.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:27.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:28.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:28.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:28.879 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:17:28.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:17:28.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:28.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:29.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:30.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:30.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:30.954 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:17:30.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:17:30.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:30.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:31.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:32.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:32.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:32.864 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:17:32.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:17:32.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:32.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:34.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:35.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:35.034 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:17:35.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:35.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:17:35.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:35.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:35.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:36.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:36.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:36.976 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:17:36.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:17:36.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:36.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:17:37.453 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1529854101,v1:127.0.0.1:6803/1529854101] [v2:127.0.0.1:6804/1529854101,v1:127.0.0.1:6805/1529854101] exists,up 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:17:37.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:17:37.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:17:37.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:17:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:17:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:17:37.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:17:37.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:17:37.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:17:37.460 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:17:37.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe' 2024-08-30T19:17:37.461 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:17:37.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRGtJmtoj8HBAA2ii2VWBF4rQLH21+98qcQA== 2024-08-30T19:17:37.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRGtJmtoj8HBAA2ii2VWBF4rQLH21+98qcQA=="}' 2024-08-30T19:17:37.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe -i td/divergent-priors/1/new.json 2024-08-30T19:17:38.018 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:17:38.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-30T19:17:38.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRGtJmtoj8HBAA2ii2VWBF4rQLH21+98qcQA== --osd-uuid 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:17:38.070 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:38.073+0000 7fe07c5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:38.122 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:38.125+0000 7fe07c5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:38.124 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:38.125+0000 7fe07c5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:38.124 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:38.125+0000 7fe07c5c07c0 -1 bdev(0x5625b0bcea80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:17:38.124 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:38.125+0000 7fe07c5c07c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-30T19:17:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-30T19:17:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:17:44.997 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:17:44.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:17:44.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:17:45.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:17:45.554 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:17:45.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:17:45.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:17:45.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:17:45.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:17:45.594 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:45.597+0000 7f110cb247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:45.609 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:45.609+0000 7f110cb247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:45.610 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:45.613+0000 7f110cb247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:17:46.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:17:46.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:17:46.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:46.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:46.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:46.857 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:46.857+0000 7f110cb247c0 -1 Falling back to public interface 2024-08-30T19:17:47.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:47.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:47.581 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:17:47.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:17:47.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:47.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:48.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:49.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:49.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:49.023 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:17:49.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:17:49.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:49.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:49.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:49.751 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:17:49.753+0000 7f110cb247c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:17:50.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:50.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:50.465 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:17:50.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:17:50.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:50.465 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:51.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:52.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:52.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:52.021 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:17:52.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:17:52.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:52.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:52.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:53.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:53.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:53.630 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:17:53.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:17:53.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:53.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:54.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:55.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:55.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:55.145 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:17:55.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:17:55.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:55.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:55.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:56.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:56.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:56.706 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:17:56.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:17:56.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:56.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:57.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:58.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:58.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:58.233 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:17:58.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:17:58.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:58.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:17:58.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:17:59.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:17:59.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:17:59.807 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:17:59.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:17:59.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:17:59.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:00.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:01.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:01.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:01.303 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:18:01.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:18:01.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:01.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:01.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:02.852 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:02.852 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:02.853 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:18:02.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:18:02.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:02.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:03.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:04.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:04.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:04.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:18:04.326 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:18:04.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:04.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:04.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:05.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:05.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:05.985 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:18:05.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:18:05.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:05.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:06.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:07.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:07.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:07.911 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:18:07.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:18:07.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:07.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:09.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:10.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:10.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:10.134 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:18:10.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:18:10.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:10.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:10.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:11.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:11.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:11.996 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:18:11.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:18:11.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:11.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:12.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:13.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:13.486 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:18:13.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:13.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:18:13.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:13.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:14.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:15.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:15.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:15.648 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:18:15.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:18:15.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:15.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:18:16.183 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/316013345,v1:127.0.0.1:6811/316013345] [v2:127.0.0.1:6812/316013345,v1:127.0.0.1:6813/316013345] exists,up 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:18:16.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:18:16.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:18:16.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:18:16.188 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:18:16.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:18:16.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:18:16.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:18:16.190 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:18:16.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 da577168-f23d-4f51-9425-556c5bc11b6b' 2024-08-30T19:18:16.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:18:16.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4GtJm6SELDRAA924Ce2ObdSvYyTIqMi0C9Q== 2024-08-30T19:18:16.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4GtJm6SELDRAA924Ce2ObdSvYyTIqMi0C9Q=="}' 2024-08-30T19:18:16.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da577168-f23d-4f51-9425-556c5bc11b6b -i td/divergent-priors/2/new.json 2024-08-30T19:18:16.893 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:18:16.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-30T19:18:16.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4GtJm6SELDRAA924Ce2ObdSvYyTIqMi0C9Q== --osd-uuid da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:18:16.959 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:16.960+0000 7fe800de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:16.961 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:16.960+0000 7fe800de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:16.963 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:16.964+0000 7fe800de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:16.963 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:16.964+0000 7fe800de37c0 -1 bdev(0x557cefe10a80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:18:16.963 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:16.964+0000 7fe800de37c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-30T19:18:23.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-30T19:18:23.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:18:23.797 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:18:23.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:18:23.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:18:24.367 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:18:24.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:18:24.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:18:24.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:18:24.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:18:24.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:18:24.408 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:24.408+0000 7f32a50447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:24.433 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:24.432+0000 7f32a50447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:24.434 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:24.436+0000 7f32a50447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:18:24.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:24.856 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:18:24.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:18:24.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:24.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:25.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:25.693 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:25.692+0000 7f32a50447c0 -1 Falling back to public interface 2024-08-30T19:18:26.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:26.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:26.288 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:18:26.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:18:26.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:26.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:26.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:27.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:27.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:27.711 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:18:27.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:18:27.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:27.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:28.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:28.668 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:18:28.668+0000 7f32a50447c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:18:29.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:29.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:29.149 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:18:29.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:18:29.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:29.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:29.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:30.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:30.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:30.647 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:18:30.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:18:30.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:30.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:31.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:32.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:32.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:32.095 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:18:32.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:18:32.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:32.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:32.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:33.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:33.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:33.618 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:18:33.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:18:33.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:33.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:34.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:35.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:35.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:35.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:18:35.057 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:18:35.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:35.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:35.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:36.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:36.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:36.480 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:18:36.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:18:36.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:36.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:36.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:37.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:37.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:37.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:18:37.974 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:18:37.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:37.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:38.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:39.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:39.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:39.405 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:18:39.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:18:39.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:39.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:39.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:40.866 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:40.866 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:40.867 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:18:40.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:18:40.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:40.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:41.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:42.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:42.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:42.304 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:18:42.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:18:42.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:42.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:43.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:44.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:44.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:44.011 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:18:44.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:18:44.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:44.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:44.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:45.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:45.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:45.616 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:18:45.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:18:45.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:45.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:46.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:47.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:47.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:47.444 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:18:47.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:18:47.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:47.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:48.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:49.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:49.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:49.303 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:18:49.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:18:49.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:49.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:50.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:51.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:51.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:51.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:18:51.071 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:18:51.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:51.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:51.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:52.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:52.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:52.796 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:18:52.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:18:52.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:52.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:53.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:18:54.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:18:54.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:18:54.627 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:18:54.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:18:54.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:18:54.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:18:55.078 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4157048286,v1:127.0.0.1:6819/4157048286] [v2:127.0.0.1:6820/4157048286,v1:127.0.0.1:6821/4157048286] exists,up da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:18:55.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:18:55.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:18:55.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:18:55.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-08-30T19:18:55.775 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-08-30T19:18:55.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-08-30T19:18:56.562 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-08-30T19:18:56.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-08-30T19:18:57.604 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-08-30T19:18:57.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-08-30T19:18:57.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=test 2024-08-30T19:18:57.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-08-30T19:18:57.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-08-30T19:18:57.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-08-30T19:18:57.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-30T19:18:58.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-08-30T19:18:58.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-08-30T19:18:59.294 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-08-30T19:18:59.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:19:00.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-08-30T19:19:00.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-08-30T19:19:01.241 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 allow_ec_overwrites to true 2024-08-30T19:19:01.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-08-30T19:19:01.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:19:01.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:19:01.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:19:01.261 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:19:01.262 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:19:01.262 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:19:01.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:19:01.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:19:01.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:19:01.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:19:01.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:19:01.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:19:01.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:19:01.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:19:01.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:19:01.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:19:01.861 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:19:01.862 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:19:01.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:19:01.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:01.862 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:19:02.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-08-30T19:19:02.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-08-30T19:19:02.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-08-30T19:19:02.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:02.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:19:02.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-08-30T19:19:02.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-08-30T19:19:02.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-08-30T19:19:02.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:02.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:19:02.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-08-30T19:19:02.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-08-30T19:19:02.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509445' 2024-08-30T19:19:02.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:02.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-08-30T19:19:02.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:02.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:19:02.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-08-30T19:19:02.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:02.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-08-30T19:19:02.356 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836502 2024-08-30T19:19:02.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-30T19:19:02.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:19:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836502 2024-08-30T19:19:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:19:03.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:19:03.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:19:04.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-08-30T19:19:04.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:04.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-08-30T19:19:04.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:04.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:19:04.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-08-30T19:19:04.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:04.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-08-30T19:19:04.199 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672977 2024-08-30T19:19:04.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-08-30T19:19:04.199 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:19:04.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-08-30T19:19:04.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:04.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:04.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-08-30T19:19:04.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:19:04.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-08-30T19:19:04.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:04.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-08-30T19:19:04.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-08-30T19:19:04.649 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509445 2024-08-30T19:19:04.650 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:05.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-08-30T19:19:05.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:19:05.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:05.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:05.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:19:05.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:19:05.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:19:05.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:19:05.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:19:05.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:19:05.607 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:19:05.607 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:19:06.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:19:06.050 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:19:06.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:06.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:06.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:19:06.634 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:19:07.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:19:07.069 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:19:07.069 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:19:07.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:19:07.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:07.070 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:19:07.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-08-30T19:19:07.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-08-30T19:19:07.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-08-30T19:19:07.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:07.249 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:19:07.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-08-30T19:19:07.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-08-30T19:19:07.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672980' 2024-08-30T19:19:07.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:07.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:19:07.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-08-30T19:19:07.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-08-30T19:19:07.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672980 2-64424509448' 2024-08-30T19:19:07.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:07.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-08-30T19:19:07.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:07.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:19:07.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-08-30T19:19:07.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:07.549 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836505 2024-08-30T19:19:07.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-08-30T19:19:07.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-30T19:19:07.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:19:07.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-08-30T19:19:07.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:07.971 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-08-30T19:19:07.971 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:07.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:19:07.974 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-08-30T19:19:07.974 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:07.975 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672980 2024-08-30T19:19:07.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-08-30T19:19:07.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-08-30T19:19:07.975 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:19:08.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-08-30T19:19:08.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:08.386 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:08.386 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-08-30T19:19:08.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:19:08.388 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-08-30T19:19:08.388 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:08.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-08-30T19:19:08.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-08-30T19:19:08.390 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509448 2024-08-30T19:19:08.390 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:08.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-08-30T19:19:08.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-08-30T19:19:08.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:19:08.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:19:08.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:19:08.809 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:19:08.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:19:08.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:19:08.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:19:08.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:19:08.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:19:08.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:19:08.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:19:08.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:19:08.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:19:08.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:19:08.915 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:19:09.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:19:09.337 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:19:09.337 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:19:09.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:19:09.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:09.338 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:19:09.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-08-30T19:19:09.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-08-30T19:19:09.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-08-30T19:19:09.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:09.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:19:09.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-08-30T19:19:09.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-08-30T19:19:09.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-08-30T19:19:09.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:09.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509450' 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-08-30T19:19:09.836 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:19:09.837 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-08-30T19:19:09.837 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:09.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-08-30T19:19:09.839 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836508 2024-08-30T19:19:09.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-08-30T19:19:09.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:19:10.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-08-30T19:19:10.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:10.234 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-08-30T19:19:10.234 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:10.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:19:10.236 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-08-30T19:19:10.236 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:10.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-08-30T19:19:10.238 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672982 2024-08-30T19:19:10.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-08-30T19:19:10.239 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:19:10.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-08-30T19:19:10.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:10.669 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-08-30T19:19:10.669 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:10.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:19:10.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-08-30T19:19:10.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:10.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-08-30T19:19:10.673 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509450 2024-08-30T19:19:10.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-30T19:19:10.674 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:11.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509450 2024-08-30T19:19:11.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:19:12.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:19:12.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:12.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509450 2024-08-30T19:19:12.542 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:19:12.543 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:12.543 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:13.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:19:13.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:19:13.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:19:13.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:19:13.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:19:13.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:13.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:14.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:19:14.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:19:14.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:19:14.052 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-08-30T19:19:14.052 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-08-30T19:19:14.484 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:19:14.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-08-30T19:19:14.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-08-30T19:19:14.502 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-08-30T19:19:14.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:19:14.875 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:19:14.875 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-30T19:19:00.317036+0000 0'0 29:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:31:15.403298+0000 0 0 2024-08-30T19:19:14.875 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:19:14.875 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:19:14.876 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:19:14.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-08-30T19:19:14.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-30T19:19:14.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-08-30T19:19:14.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-08-30T19:19:14.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-30T19:19:14.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-08-30T19:19:14.891 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:14.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-08-30T19:19:15.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:15.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-08-30T19:19:15.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:15.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-08-30T19:19:15.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:15.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-08-30T19:19:15.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:15.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-08-30T19:19:16.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:16.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-08-30T19:19:16.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:16.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-08-30T19:19:16.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:16.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-08-30T19:19:16.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:16.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-08-30T19:19:17.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:17.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-08-30T19:19:17.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:17.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-08-30T19:19:17.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:17.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-08-30T19:19:18.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:18.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-08-30T19:19:18.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:18.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-08-30T19:19:18.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:18.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-08-30T19:19:18.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:18.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-08-30T19:19:19.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:19.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-08-30T19:19:19.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:19.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-08-30T19:19:19.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:19.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-08-30T19:19:19.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:19.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-08-30T19:19:19.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:19.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-08-30T19:19:20.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:20.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-08-30T19:19:20.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:20.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-08-30T19:19:20.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:20.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-08-30T19:19:21.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:21.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-08-30T19:19:21.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:21.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-08-30T19:19:21.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:21.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-08-30T19:19:21.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:21.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-08-30T19:19:22.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:22.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-08-30T19:19:22.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:22.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-08-30T19:19:22.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:22.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-08-30T19:19:22.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:22.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-08-30T19:19:23.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:23.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-08-30T19:19:23.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:23.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-08-30T19:19:23.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:23.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-08-30T19:19:23.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:23.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-08-30T19:19:24.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:24.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-08-30T19:19:24.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:24.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-08-30T19:19:24.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:24.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-08-30T19:19:24.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:24.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-08-30T19:19:25.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:25.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-08-30T19:19:25.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:25.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-08-30T19:19:25.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:25.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-08-30T19:19:26.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:26.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-08-30T19:19:26.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:26.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-08-30T19:19:26.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:26.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-08-30T19:19:27.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:27.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-08-30T19:19:27.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:27.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-08-30T19:19:27.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:27.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-08-30T19:19:27.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:27.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-08-30T19:19:28.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:28.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-08-30T19:19:28.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:28.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-08-30T19:19:28.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:28.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-08-30T19:19:28.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:28.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-08-30T19:19:29.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:29.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-08-30T19:19:29.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:29.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-08-30T19:19:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-08-30T19:19:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:29.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-08-30T19:19:30.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:30.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-08-30T19:19:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-08-30T19:19:30.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:30.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-08-30T19:19:30.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:30.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-08-30T19:19:31.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:31.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-08-30T19:19:31.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:31.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-08-30T19:19:31.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:31.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-08-30T19:19:31.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:31.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-08-30T19:19:32.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:32.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-08-30T19:19:32.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:32.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-08-30T19:19:32.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:32.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-08-30T19:19:32.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:32.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-08-30T19:19:33.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:33.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-08-30T19:19:33.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:33.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-08-30T19:19:33.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:33.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-08-30T19:19:33.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:33.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-08-30T19:19:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:34.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-08-30T19:19:34.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:34.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-08-30T19:19:34.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:34.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-08-30T19:19:34.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:34.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-08-30T19:19:35.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:35.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-08-30T19:19:35.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:35.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-08-30T19:19:35.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:35.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-08-30T19:19:35.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:35.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-08-30T19:19:36.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:36.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-08-30T19:19:36.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:36.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-08-30T19:19:36.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:36.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-08-30T19:19:36.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:36.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-08-30T19:19:37.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:37.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-08-30T19:19:37.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:37.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-08-30T19:19:37.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:37.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-08-30T19:19:38.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:38.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-08-30T19:19:38.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:38.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-08-30T19:19:38.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:38.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-08-30T19:19:38.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:38.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-08-30T19:19:39.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:39.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-08-30T19:19:39.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:39.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-08-30T19:19:39.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:39.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-08-30T19:19:39.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:39.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-08-30T19:19:40.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:40.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-08-30T19:19:40.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:40.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-08-30T19:19:40.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-30T19:19:40.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-08-30T19:19:40.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:19:40.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-08-30T19:19:40.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:19:40.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:19:40.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:19:40.842 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:19:40.842 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:19:40.842 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:19:40.843 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:19:40.843 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:19:40.843 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:19:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:19:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:19:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:19:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:19:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:19:40.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:19:41.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:19:41.352 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:19:41.353 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:19:41.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:19:41.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:41.353 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:19:41.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-08-30T19:19:41.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-08-30T19:19:41.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-08-30T19:19:41.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:41.526 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:19:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-08-30T19:19:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-08-30T19:19:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672991' 2024-08-30T19:19:41.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:19:41.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:19:41.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-08-30T19:19:41.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-08-30T19:19:41.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672991 2-64424509459' 2024-08-30T19:19:41.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:41.858 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-08-30T19:19:41.858 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:41.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:19:41.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-08-30T19:19:41.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:41.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-08-30T19:19:41.861 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836516 2024-08-30T19:19:41.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-08-30T19:19:41.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:19:42.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-08-30T19:19:42.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:42.296 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-08-30T19:19:42.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:42.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:19:42.299 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-08-30T19:19:42.299 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:42.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-08-30T19:19:42.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-08-30T19:19:42.301 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672991 2024-08-30T19:19:42.301 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:19:42.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-08-30T19:19:42.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:19:42.718 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-08-30T19:19:42.718 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:19:42.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:19:42.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-08-30T19:19:42.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:19:42.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-08-30T19:19:42.722 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509459 2024-08-30T19:19:42.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-30T19:19:42.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:43.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509459 2024-08-30T19:19:43.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:19:44.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:19:44.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:19:44.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-30T19:19:44.605 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:19:44.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:44.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:45.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:19:45.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:19:45.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:19:45.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:19:45.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:19:45.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:19:45.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:19:46.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:19:46.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:19:46.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:19:46.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-08-30T19:19:46.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-30T19:19:46.161 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-30T19:19:46.162 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-30T19:19:46.162 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-30T19:19:46.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-08-30T19:19:46.626 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-08-30T19:19:46.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-08-30T19:19:46.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:19:47.032 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:19:47.033 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-30T19:19:00.317036+0000 29'100 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T23:31:15.403298+0000 0 0 2024-08-30T19:19:47.033 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:19:47.033 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:19:47.033 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:19:47.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:19:47.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-08-30T19:19:47.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:19:47.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:19:47.049 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:19:47.049 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:19:47.050 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:19:47.050 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:19:47.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-08-30T19:19:47.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-30T19:19:47.146 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:19:47.146 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:19:47.146 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:19:47.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:19:47.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-08-30T19:19:47.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:19:47.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:19:47.158 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:19:47.159 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:19:47.159 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:19:47.159 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:19:47.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-08-30T19:19:47.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-30T19:19:47.267 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:19:47.267 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-30T19:19:47.267 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:19:47.278 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent object 2024-08-30T19:19:47.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-08-30T19:19:47.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-08-30T19:19:47.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-08-30T19:19:48.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-08-30T19:19:48.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-08-30T19:19:48.623 INFO:tasks.workunit.client.0.smithi079.stdout:created pool test snap snap1 2024-08-30T19:19:48.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-08-30T19:19:48.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-08-30T19:19:48.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-08-30T19:19:58.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-08-30T19:19:58.633 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-08-30T19:19:58.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-08-30T19:19:58.633 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-08-30T19:19:58.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:19:59.043 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:19:59.043 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-08-30T19:19:00.317036+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:02:07.198863+0000 0 0 2024-08-30T19:19:59.043 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:19:59.043 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:19:59.044 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:19:59.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-08-30T19:19:59.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:19:59.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:19:59.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:19:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:19:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:19:59.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:19:59.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-30T19:19:59.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-08-30T19:19:59.796 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:19:59.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-30T19:19:59.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-08-30T19:20:00.430 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:20:00.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-30T19:20:00.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-08-30T19:20:01.040 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:20:01.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-08-30T19:20:01.649 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-08-30T19:20:01.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:01.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-08-30T19:20:02.282 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-08-30T19:20:02.299 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-08-30T19:20:02.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-08-30T19:20:02.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:20:02.724 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:20:02.724 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-08-30T19:19:00.317036+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:02:07.198863+0000 0 0 2024-08-30T19:20:02.724 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:20:02.724 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:20:02.724 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:20:02.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:02.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-08-30T19:20:02.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:20:02.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:20:02.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:20:02.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:20:02.746 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:20:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:20:02.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:20:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:20:02.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:20:02.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:20:02.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:20:02.777 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:02.778+0000 7faeb0aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:02.780 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:02.782+0000 7faeb0aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:02.781 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:02.782+0000 7faeb0aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:20:03.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:03.170 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:20:03.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:20:03.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:03.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:20:03.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:04.032 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:04.034+0000 7faeb0aff7c0 -1 Falling back to public interface 2024-08-30T19:20:04.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:04.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:04.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:20:04.629 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:20:04.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:04.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:20:05.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:06.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:06.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:06.064 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:20:06.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:20:06.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:06.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:20:06.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:07.374 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:07.378+0000 7faeb0aff7c0 -1 osd.0 29 log_to_monitors true 2024-08-30T19:20:07.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:07.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:07.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:20:07.584 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:20:07.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:07.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:20:08.152 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3830116600,v1:127.0.0.1:6803/3830116600] [v2:127.0.0.1:6804/3830116600,v1:127.0.0.1:6805/3830116600] exists,up 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:20:08.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:20:08.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:20:08.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:20:08.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:20:08.156 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:20:08.156 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:20:08.156 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:20:08.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:20:08.158 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:20:08.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:20:08.158 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-30T19:20:08.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:20:08.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:20:08.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:20:08.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:20:08.197 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:08.198+0000 7f8cbabcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:08.222 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:08.226+0000 7f8cbabcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:08.223 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:08.226+0000 7f8cbabcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:08.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:20:08.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:20:08.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:20:08.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:20:08.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:20:08.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:08.626 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:20:08.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:20:08.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:08.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:20:09.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:09.256 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:09.258+0000 7f8cbabcf7c0 -1 Falling back to public interface 2024-08-30T19:20:10.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:10.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:10.050 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:20:10.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:20:10.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:10.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:20:10.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:11.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:11.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:11.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:20:11.508 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:20:11.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:11.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:20:11.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:20:12.831 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:12.834+0000 7f8cbabcf7c0 -1 osd.2 29 log_to_monitors true 2024-08-30T19:20:12.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:20:12.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:20:12.917 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:20:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:20:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:20:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:20:13.368 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/1568983399,v1:127.0.0.1:6811/1568983399] [v2:127.0.0.1:6812/1568983399,v1:127.0.0.1:6813/1568983399] exists,up da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:20:13.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:20:13.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:20:13.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:20:13.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:13.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-08-30T19:20:13.904 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-08-30T19:20:13.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:13.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-08-30T19:20:14.465 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-08-30T19:20:14.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-08-30T19:20:19.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-08-30T19:20:19.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-08-30T19:20:19.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-08-30T19:20:19.485 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-08-30T19:20:19.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:20:19.902 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:20:19.902 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-30T19:20:15.285961+0000 29'100 43:126 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:56:56.019302+0000 0 0 2024-08-30T19:20:19.902 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:20:19.902 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:20:19.902 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:20:19.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-08-30T19:20:20.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:20:20.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-08-30T19:20:20.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:20:20.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:20:20.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:20:20.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:20:20.060 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:20:20.060 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:20:20.060 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:20:20.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:20:20.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:20:20.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:20:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:20:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:20:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:20:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:20:20.137 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:20:20.567 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:20:20.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789957 2024-08-30T19:20:20.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789957 2024-08-30T19:20:20.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789957' 2024-08-30T19:20:20.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:20:20.754 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:20:20.877 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T19:20:20.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:20:20.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:20:20.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:20:20.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:20:20.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:20:21.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724548 2024-08-30T19:20:21.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724548 2024-08-30T19:20:21.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-08-30T19:20:21.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:20:21.032 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-158913789957 2024-08-30T19:20:21.032 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:20:21.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:20:21.033 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-158913789957 2024-08-30T19:20:21.034 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:20:21.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789957 2024-08-30T19:20:21.035 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 158913789957 2024-08-30T19:20:21.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-08-30T19:20:21.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:20:21.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789955 -lt 158913789957 2024-08-30T19:20:21.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:20:22.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:20:22.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:20:22.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789957 -lt 158913789957 2024-08-30T19:20:22.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:20:22.857 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724548 2024-08-30T19:20:22.857 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:20:22.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:20:22.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:20:22.859 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724548 2024-08-30T19:20:22.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724548 2024-08-30T19:20:22.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-08-30T19:20:22.860 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 167503724548 2024-08-30T19:20:22.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:20:23.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724548 2024-08-30T19:20:23.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:20:23.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:23.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:23.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:20:23.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:23.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:23.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:23.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:23.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:23.869 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:23.869 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:24.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:24.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:24.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:24.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:24.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:24.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:20:24.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:20:24.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:20:24.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:24.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:20:24.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:24.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:24.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:24.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:24.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:24.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:24.986 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:24.986 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:25.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:25.412 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:25.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:25.413 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:25.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:25.976 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:25.976 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:26.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:26.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:26.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:20:26.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:26.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:20:26.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:26.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:26.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:26.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:26.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:26.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:26.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:26.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:27.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:27.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:27.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:27.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:27.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:27.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:27.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:27.685 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:28.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:28.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:28.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:20:28.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:28.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:20:28.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:28.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:28.625 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:28.625 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:28.625 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:28.625 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:28.625 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:28.626 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:29.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:29.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:29.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:29.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:29.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:29.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:29.581 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:30.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:30.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:30.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:20:30.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:30.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:20:30.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:30.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:30.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:30.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:30.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:30.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:30.947 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:30.947 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:31.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:31.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:31.363 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:31.363 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:31.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:31.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:31.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:31.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:31.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:31.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:31.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:31.903 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:31.903 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:32.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:32.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:32.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:20:32.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:32.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:20:34.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:34.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:34.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:34.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:34.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:34.057 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:34.057 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:34.057 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:34.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:34.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:34.470 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:34.470 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:35.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:35.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:35.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:35.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:35.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:35.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:35.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:35.022 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:35.022 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:35.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:35.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:35.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:20:35.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:35.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:20:38.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:38.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:38.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:38.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:38.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:38.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:38.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:38.781 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:39.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:39.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:39.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:39.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:39.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:39.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:39.755 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:39.755 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:40.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:40.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:40.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:20:40.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:40.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:20:46.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:46.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:46.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:46.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:46.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:46.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:46.759 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:46.759 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:47.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:47.184 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:47.185 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:47.185 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:47.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:47.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:47.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:47.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:47.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:47.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:47.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:47.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:47.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:48.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:48.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:48.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:20:48.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:20:48.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:20:55.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:20:55.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:20:55.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:20:55.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:20:55.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:20:55.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:20:55.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:20:55.649 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:20:56.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:20:56.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:20:56.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:20:56.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:20:56.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:20:56.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:20:57.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:20:57.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:20:57.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:20:57.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:20:57.785 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "028bb826-4978-4e5c-8498-010aa3a0a542", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:20:57.789607+0000", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:20:57.786 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:20:57.787 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:20:57.788 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:20:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.789 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:20:57.790 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.791 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:20:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:20:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.792 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:20:57.793 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:16:57.437407Z", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:16:57.437407Z", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:20:57.794 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.795 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:20:57.796 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 43, 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:20:57.797 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:16:58.328519+0000", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:20:14.905622+0000", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:20:13.206644+0000", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:20:14.296499+0000", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:20:57.798 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 10, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:20:57.799 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:18:58.865537+0000", 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 16389, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 1, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:20:57.800 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:20:57.801 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "30", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "pool", 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 1, 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 30, 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [ 2024-08-30T19:20:57.802 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "snapid": 1, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "stamp": "2024-08-30T19:19:48.331717+0000", 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "snap1" 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:20:57.803 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "myprofile", 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:20:57.804 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 8192, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:20:57.805 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {} 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "3f69e6db-716e-4819-87c9-5d2444ea98b7", 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-08-30T19:20:57.806 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 37, 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 42, 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:20:57.807 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.808 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.809 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:20:57.810 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3830116600 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3830116600", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3830116600", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3830116600", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3830116600", 2024-08-30T19:20:57.811 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9ec6ed1f-9817-4f59-bd80-a4edf1e214fe", 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:20:57.812 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 25, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.813 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.814 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.815 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.816 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/316013345", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/316013345", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/316013345", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/316013345", 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:20:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "da577168-f23d-4f51-9425-556c5bc11b6b", 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 39, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 33, 2024-08-30T19:20:57.818 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.819 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:20:57.820 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:20:57.821 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1568983399 2024-08-30T19:20:57.822 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1568983399", 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1568983399", 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1568983399", 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1568983399", 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:20:57.823 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:19:59.530613+0000", 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:17:36.093335+0000", 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:20:00.231716+0000", 2024-08-30T19:20:57.824 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:18:15.207668+0000", 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:20:00.834502+0000", 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:20:57.825 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:18:53.833689+0000", 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:20:57.826 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "myprofile": { 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-device-class": "", 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-failure-domain": "osd", 2024-08-30T19:20:57.827 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-num-failure-domains": "0", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-osds-per-failure-domain": "0", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-root": "default", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "jerasure-per-chunk-alignment": "false", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "1", 2024-08-30T19:20:57.828 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van", 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "w": "8" 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:20:57.829 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3830116600,v1:127.0.0.1:6805/3830116600]", 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:20:57.830 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:20:57.831 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:20:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:20:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:20:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:20:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:20:57.832 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:17:05.560878Z", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:20:57.833 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3830116600,v1:127.0.0.1:6803/3830116600]", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:20:57.834 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3830116600,v1:127.0.0.1:6809/3830116600]", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3830116600,v1:127.0.0.1:6807/3830116600]", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:20:57.835 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/316013345,v1:127.0.0.1:6813/316013345]", 2024-08-30T19:20:57.836 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:20:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:20:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:20:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:20:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:20:57.837 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:20:57.838 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:17:44.245629Z", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:20:57.839 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/316013345,v1:127.0.0.1:6811/316013345]", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/316013345,v1:127.0.0.1:6817/316013345]", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/316013345,v1:127.0.0.1:6815/316013345]", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:20:57.840 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.841 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1568983399,v1:127.0.0.1:6813/1568983399]", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:20:57.842 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.843 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:18:23.119450Z", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1568983399,v1:127.0.0.1:6811/1568983399]", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1568983399,v1:127.0.0.1:6817/1568983399]", 2024-08-30T19:20:57.844 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1568983399,v1:127.0.0.1:6815/1568983399]", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:20:57.845 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 37 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.846 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 10 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 39 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 43, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:20:57.847 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:20:57.848 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.849 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:20:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:20:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:20:57.850 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.851 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:20:57.852 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.853 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:20:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:20:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:20:57.854 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:20:57.855 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.856 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:20:57.857 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:20:57.858 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:20:57.859 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.860 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:20:57.861 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.862 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:20:57.863 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:20:57.864 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 1, 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "test", 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_chooseleaf_tries", 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 5 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.865 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_choose_tries", 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 100 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_indep", 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.866 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:20:57.867 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:20:57.868 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:16:58:304133+0000", 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:20:57.869 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:20:57.870 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.871 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62667, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:20:57.872 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:20:57.873 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:20:57.874 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:20:57.875 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-08-30T19:20:57.876 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83692, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2164, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:20:57.877 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489108, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036846592, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2215936, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1550914, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.878 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.879 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83692, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2164, 2024-08-30T19:20:57.880 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489108, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036846592, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2215936, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1550914, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416932 2024-08-30T19:20:57.881 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:20:57.882 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62667, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:20:57.883 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:20:57.884 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:20:57.885 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1241088, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1241088, 2024-08-30T19:20:57.886 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:20:57.887 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 158913789964 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672994 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 167503724555 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:20:57.888 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2147483647, 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.889 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:20:57.890 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:19:55.786926+0000", 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:20:57.891 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 108, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 423, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 581, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:20:57.892 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:20:57.893 INFO:tasks.workunit.client.0.smithi079.stderr:report 3661835607 2024-08-30T19:20:57.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:20:57.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:20:57.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-08-30T19:20:57.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:20:57.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:20:57.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:20:57.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:20:57.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-30T19:20:57.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-08-30T19:20:59.485 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:20:59.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:20:59.786 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:20:59.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-30T19:20:59.789 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:20:59.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:20:59.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:20:59.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-30T19:20:59.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:20:59.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:20:59.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:20:59.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:20:59.839 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:59.841+0000 7fbc8a0d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:59.841 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:59.845+0000 7fbc8a0d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:20:59.842 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:20:59.845+0000 7fbc8a0d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:21:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:21:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:21:00.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:21:00.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:21:00.676 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3830116600,v1:127.0.0.1:6803/3830116600] [v2:127.0.0.1:6804/3830116600,v1:127.0.0.1:6805/3830116600] exists,up 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:21:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:21:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:21:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:21:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:21:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-08-30T19:21:00.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:21:00.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:21:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:21:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:21:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-30T19:21:00.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:21:00.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-08-30T19:21:00.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-08-30T19:21:00.879 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:00.881+0000 7fbc8a0d97c0 -1 Falling back to public interface 2024-08-30T19:21:02.647 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-08-30T19:21:02.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-30T19:21:02.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:21:02.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:21:02.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:21:02.939 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:21:02.939 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:21:02.939 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:21:02.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-30T19:21:02.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:21:02.942 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:21:02.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:21:02.942 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-30T19:21:02.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:21:02.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:21:02.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:21:02.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:21:02.983 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:02.985+0000 7f33e0c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:02.986 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:02.989+0000 7f33e0c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:02.988 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:02.989+0000 7f33e0c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:03.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:21:03.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:21:03.367 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:21:03.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:21:03.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:21:03.689 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:03.693+0000 7fbc8a0d97c0 -1 osd.0 43 log_to_monitors true 2024-08-30T19:21:03.932 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/1568983399,v1:127.0.0.1:6811/1568983399] [v2:127.0.0.1:6812/1568983399,v1:127.0.0.1:6813/1568983399] exists,up da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-30T19:21:03.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-08-30T19:21:04.216 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:04.217+0000 7f33e0c3a7c0 -1 Falling back to public interface 2024-08-30T19:21:05.905 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-08-30T19:21:06.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:21:06.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-30T19:21:06.223 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:21:06.223 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:21:06.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:21:06.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:21:06.224 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:21:06.294 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:21:06.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:21:06.777 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:21:06.777 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-08-30T19:21:06.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:21:06.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:21:06.778 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:21:06.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495619 2024-08-30T19:21:06.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495619 2024-08-30T19:21:06.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495619' 2024-08-30T19:21:06.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:21:06.975 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:21:07.088 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T19:21:07.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:21:07.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:21:07.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:21:07.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:21:07.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:21:07.202 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:07.205+0000 7f33e0c3a7c0 -1 osd.2 43 log_to_monitors true 2024-08-30T19:21:12.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364803 2024-08-30T19:21:12.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364803 2024-08-30T19:21:12.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-08-30T19:21:12.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:21:12.270 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495619 2024-08-30T19:21:12.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:21:12.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:21:12.275 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495619 2024-08-30T19:21:12.275 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:21:12.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495619 2024-08-30T19:21:12.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-08-30T19:21:12.276 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 197568495619 2024-08-30T19:21:12.277 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:21:12.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495620 -lt 197568495619 2024-08-30T19:21:12.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:21:12.816 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:21:12.816 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-214748364803 2024-08-30T19:21:12.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:21:12.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-214748364803 2024-08-30T19:21:12.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:21:12.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364803 2024-08-30T19:21:12.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-08-30T19:21:12.820 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 214748364803 2024-08-30T19:21:12.820 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:21:13.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 214748364803 2024-08-30T19:21:13.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:21:14.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:21:14.284 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:21:14.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364804 -lt 214748364803 2024-08-30T19:21:14.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:21:14.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:14.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:15.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:21:15.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:15.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:15.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:15.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:15.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:15.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:15.287 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:15.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:15.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:15.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:15.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:16.277 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-30T19:21:16.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:16.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:16.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:16.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:16.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:16.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:16.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:16.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:16.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:16.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:16.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:16.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:17.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:17.328 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:17.328 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:17.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:17.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:17.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-30T19:21:17.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:17.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-30T19:21:18.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:18.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:18.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:18.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:18.496 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:18.497 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:18.497 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:19.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:19.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:19.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:19.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:19.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:19.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:19.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:19.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:19.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:19.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:19.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:19.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-30T19:21:19.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:19.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-30T19:21:20.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:20.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:20.005 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:20.005 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:20.005 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:20.005 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:20.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:20.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:20.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:20.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:20.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:20.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:20.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:20.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:20.984 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-30T19:21:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-30T19:21:22.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:22.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:22.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:22.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:22.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:22.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:22.373 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:22.373 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:22.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:22.777 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:22.778 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:22.778 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:23.331 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:23.331 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:23.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:23.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:23.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-30T19:21:23.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:23.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-30T19:21:25.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:25.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:25.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:25.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:25.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:25.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:25.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:25.489 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:25.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:25.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:25.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:25.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:26.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:26.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:26.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:26.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:27.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:27.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:27.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-30T19:21:27.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:27.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-30T19:21:30.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:30.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:30.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:30.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:30.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:30.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:30.261 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:30.261 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:30.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:30.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:30.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:30.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:31.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:31.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:31.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:31.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:31.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:31.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-30T19:21:31.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:31.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-30T19:21:38.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:38.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:38.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:38.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:38.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:38.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:38.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:38.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:38.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:38.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:38.613 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:38.613 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:39.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:39.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:39.157 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:39.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:39.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:39.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-30T19:21:39.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-30T19:21:39.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-30T19:21:47.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-30T19:21:47.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:21:47.050 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:21:47.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:21:47.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:21:47.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:21:47.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:21:47.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:21:47.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-30T19:21:47.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:21:47.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:21:47.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:21:48.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-30T19:21:48.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-30T19:21:48.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-30T19:21:48.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-30T19:21:48.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-30T19:21:48.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-30T19:21:48.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-30T19:21:48.033 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-30T19:21:48.033 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-30T19:21:48.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-30T19:21:48.582 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-30T19:21:48.582 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-30T19:21:48.582 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-30T19:21:49.143 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:21:49.143 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "028bb826-4978-4e5c-8498-010aa3a0a542", 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.1.1-235-g1ef86450", 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-08-30T19:21:49.147824+0000", 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-08-30T19:21:49.144 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.145 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.146 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.147 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.148 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-08-30T19:21:49.149 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:16:57.437407Z", 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:16:57.437407Z", 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-08-30T19:21:49.150 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-08-30T19:21:49.151 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.152 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.153 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-08-30T19:21:49.154 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 51, 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "09ccc235-e650-4ecd-972e-85dae08bf0c0", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-08-30T19:16:58.328519+0000", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:21:08.119614+0000", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-08-30T19:21:07.650621+0000", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-08-30T19:20:14.296499+0000", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-08-30T19:21:49.155 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 12, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-08-30T19:21:49.156 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-08-30T19:18:58.865537+0000", 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 16389, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-08-30T19:21:49.157 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 1, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-08-30T19:21:49.158 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.159 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "30", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "pool", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 1, 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 30, 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [ 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "snapid": 1, 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "stamp": "2024-08-30T19:19:48.331717+0000", 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "snap1" 2024-08-30T19:21:49.160 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-08-30T19:21:49.161 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "myprofile", 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-08-30T19:21:49.162 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 8192, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {} 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:21:49.163 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "3f69e6db-716e-4819-87c9-5d2444ea98b7", 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 37, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 43, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 46, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 50, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 45, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.164 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.165 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.166 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-08-30T19:21:49.167 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.168 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 811824041 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/811824041", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/811824041", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/811824041", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/811824041", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.169 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9ec6ed1f-9817-4f59-bd80-a4edf1e214fe", 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 25, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:21:49.170 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.171 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.172 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.173 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.174 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.175 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.176 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.176 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:21:49.176 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 316013345 2024-08-30T19:21:49.176 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.176 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/316013345", 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/316013345", 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/316013345", 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/316013345", 2024-08-30T19:21:49.177 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "da577168-f23d-4f51-9425-556c5bc11b6b", 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-08-30T19:21:49.178 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 39, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 43, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 50, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 49, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-08-30T19:21:49.179 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.180 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.181 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-08-30T19:21:49.182 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.183 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.184 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-08-30T19:21:49.185 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3356765838 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/3356765838", 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/3356765838", 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3356765838", 2024-08-30T19:21:49.186 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3356765838", 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:21:04.124595+0000", 2024-08-30T19:21:49.187 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:17:36.093335+0000", 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:20:00.231716+0000", 2024-08-30T19:21:49.188 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:18:15.207668+0000", 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-08-30T19:21:07.458175+0000", 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540701547738038271, 2024-08-30T19:21:49.189 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-08-30T19:18:53.833689+0000", 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:21:49.190 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "myprofile": { 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-device-class": "", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-failure-domain": "osd", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-num-failure-domains": "0", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-osds-per-failure-domain": "0", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-root": "default", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "jerasure-per-chunk-alignment": "false", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "1", 2024-08-30T19:21:49.191 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van", 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "w": "8" 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-08-30T19:21:49.192 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:21:49.193 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/811824041,v1:127.0.0.1:6805/811824041]", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-30T19:21:49.194 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:17:05.560878Z", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:21:49.195 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/811824041,v1:127.0.0.1:6803/811824041]", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/811824041,v1:127.0.0.1:6809/811824041]", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/811824041,v1:127.0.0.1:6807/811824041]", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:21:49.196 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:21:49.197 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/316013345,v1:127.0.0.1:6813/316013345]", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:21:49.198 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:17:44.245629Z", 2024-08-30T19:21:49.199 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/316013345,v1:127.0.0.1:6811/316013345]", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/316013345,v1:127.0.0.1:6817/316013345]", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/316013345,v1:127.0.0.1:6815/316013345]", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:21:49.200 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:21:49.201 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/3356765838,v1:127.0.0.1:6813/3356765838]", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-30T19:21:49.202 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "0", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-08-30T19:18:23.119450Z", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QDP6", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-30T19:21:49.203 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "ubuntu", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "22.04", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/3356765838,v1:127.0.0.1:6811/3356765838]", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3356765838,v1:127.0.0.1:6817/3356765838]", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3356765838,v1:127.0.0.1:6815/3356765838]", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.15.0-119-generic", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "2097148", 2024-08-30T19:21:49.204 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32746820", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 10, 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [ 2024-08-30T19:21:49.205 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "floor": 26 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 46 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.206 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 10 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 50 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 51, 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-08-30T19:21:49.207 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.208 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.209 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-08-30T19:21:49.210 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-08-30T19:21:49.211 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.212 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:21:49.213 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.214 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-08-30T19:21:49.215 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:21:49.216 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.217 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-08-30T19:21:49.218 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.219 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:21:49.220 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.221 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 1, 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "test", 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_chooseleaf_tries", 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 5 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.222 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_choose_tries", 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 100 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_indep", 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-08-30T19:21:49.223 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-08-30T19:21:49.224 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-08-30T19:21:49.225 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-08-30T19:21:49.226 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-08-30T19:16:58:304133+0000", 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-08-30T19:21:49.227 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.228 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-08-30T19:21:49.229 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-08-30T19:21:49.230 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62667, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:21:49.231 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:21:49.232 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:21:49.233 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:21:49.234 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:21:49.235 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 85120, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2292, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 15, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 82736, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314487680, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322035384320, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:21:49.236 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2347008, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1605822, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 16263, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 84721785 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:21:49.237 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:21:49.238 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-08-30T19:21:49.239 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 85120, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2292, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 15, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 82736, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314487680, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-08-30T19:21:49.240 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322035384320, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2347008, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1605822, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 16263, 2024-08-30T19:21:49.241 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 84721785 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-08-30T19:21:49.242 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.243 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 62667, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-08-30T19:21:49.244 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-08-30T19:21:49.245 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-08-30T19:21:49.246 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-08-30T19:21:49.247 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-08-30T19:21:49.247 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-08-30T19:21:49.247 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-08-30T19:21:49.247 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-08-30T19:21:49.247 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-08-30T19:21:49.248 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-08-30T19:21:49.249 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-08-30T19:21:49.250 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1241088, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1241088, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-08-30T19:21:49.251 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 197568495627 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672994 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:21:49.252 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 214748364811 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-08-30T19:21:49.253 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2147483647, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:21:49.254 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.255 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-08-30T19:21:49.256 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-08-30T19:19:55.786926+0000", 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 134, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 515, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 706, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-08-30T19:21:49.257 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:21:49.258 INFO:tasks.workunit.client.0.smithi079.stderr:report 3819268598 2024-08-30T19:21:49.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-30T19:21:49.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-08-30T19:21:49.258 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-08-30T19:21:49.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:21:49.576 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:21:49.576 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 no scrub is scheduled 0 0 2024-08-30T19:21:49.576 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:21:49.576 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:21:49.576 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:21:49.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:21:49.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-08-30T19:21:49.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-30T19:21:49.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-30T19:21:49.592 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:21:49.593 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:21:49.593 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:21:49.593 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.0.asok 2024-08-30T19:21:49.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-08-30T19:21:49.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.0.asok set_recovery_delay 100000 2024-08-30T19:21:49.693 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:21:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-08-30T19:21:49.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.2.asok 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-08-30T19:21:49.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.2.asok set_recovery_delay 100000 2024-08-30T19:21:49.817 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:21:49.830 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-08-30T19:21:49.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-08-30T19:21:49.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:21:50.224 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:21:50.224 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 no scrub is scheduled 0 0 2024-08-30T19:21:50.224 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:21:50.224 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:21:50.225 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:21:50.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-08-30T19:21:50.934 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-08-30T19:21:50.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:21:50.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:21:50.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:21:50.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:21:50.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:21:50.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:21:50.957 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:21:50.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:21:50.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:21:50.957 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:21:50.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:21:50.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:21:50.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:21:50.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:21:50.997 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:50.996+0000 7f1d11ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:51.010 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:51.012+0000 7f1d11ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:51.012 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:51.012+0000 7f1d11ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:21:51.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-30T19:21:51.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-08-30T19:21:52.032 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:52.032+0000 7f1d11ff57c0 -1 Falling back to public interface 2024-08-30T19:21:54.820 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:21:54.820+0000 7f1d11ff57c0 -1 osd.1 30 log_to_monitors true 2024-08-30T19:21:56.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-08-30T19:21:56.405 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-08-30T19:21:56.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:21:56.809 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:21:56.809 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:33:16.750220+0000 0 0 2024-08-30T19:21:56.809 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:21:56.809 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:21:56.809 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:21:56.825 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27261/ceph-osd.1.asok 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-08-30T19:21:56.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27261/ceph-osd.1.asok set_recovery_delay 100000 2024-08-30T19:21:56.939 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-30T19:21:56.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-08-30T19:21:57.586 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:21:57.606 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:21:57.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:21:57.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:21:57.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:21:58.288 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/811824041,v1:127.0.0.1:6803/811824041] [v2:127.0.0.1:6804/811824041,v1:127.0.0.1:6805/811824041] exists,up 3f69e6db-716e-4819-87c9-5d2444ea98b7 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:21:58.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:21:58.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:21:58.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:21:58.290 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:21:58.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:21:58.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:21:58.773 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/4180061341,v1:127.0.0.1:6819/4180061341] [v2:127.0.0.1:6820/4180061341,v1:127.0.0.1:6821/4180061341] exists,up 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:21:58.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:21:58.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:21:59.201 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/3356765838,v1:127.0.0.1:6811/3356765838] [v2:127.0.0.1:6812/3356765838,v1:127.0.0.1:6813/3356765838] exists,up da577168-f23d-4f51-9425-556c5bc11b6b 2024-08-30T19:21:59.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:21:59.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:21:59.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:21:59.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:21:59.619 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:21:59.619 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:33:16.750220+0000 0 0 2024-08-30T19:21:59.619 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:21:59.619 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:21:59.619 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:21:59.634 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-08-30T19:21:59.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-08-30T19:21:59.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:22:00.040 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:22:00.040 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:33:16.750220+0000 0 0 2024-08-30T19:22:00.040 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:22:00.040 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:22:00.041 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:22:00.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-08-30T19:22:00.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-08-30T19:22:00.189 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-08-30T19:22:00.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:22:00.577 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:22:00.577 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:33:16.750220+0000 0 0 2024-08-30T19:22:00.577 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:22:00.577 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:22:00.577 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:22:00.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-08-30T19:22:00.593 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:22:00.593 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:22:00.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:22:00.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:22:00.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:22:00.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:22:00.700 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-08-30T19:22:00.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-08-30T19:22:00.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:22:01.098 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:22:01.099 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-30T19:21:09.240129+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-08-31T21:33:16.750220+0000 0 0 2024-08-30T19:22:01.099 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:22:01.099 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:22:01.099 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:22:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-08-30T19:22:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-30T19:22:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:22:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:22:01.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-30T19:22:01.117 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:22:01.117 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:22:01.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:22:01.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:22:01.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-30T19:22:01.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:22:01.119 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09ccc235-e650-4ecd-972e-85dae08bf0c0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27261/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:22:01.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:22:01.157 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:01.160+0000 7fc727f967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:01.159 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:01.160+0000 7fc727f967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:01.160 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:01.160+0000 7fc727f967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:22:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:22:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:22:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:22:01.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:22:01.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:01.588 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:22:01.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:22:01.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:01.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:22:02.017 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/4180061341,v1:127.0.0.1:6819/4180061341] [v2:127.0.0.1:6820/4180061341,v1:127.0.0.1:6821/4180061341] exists,up 9ec6ed1f-9817-4f59-bd80-a4edf1e214fe 2024-08-30T19:22:02.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:22:02.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:22:02.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:22:02.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-08-30T19:22:02.245 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:02.244+0000 7fc727f967c0 -1 Falling back to public interface 2024-08-30T19:22:05.477 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:05.480+0000 7fc727f967c0 -1 osd.1 57 log_to_monitors true 2024-08-30T19:22:22.019 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-08-30T19:22:22.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-08-30T19:22:22.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:22:22.431 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:22:22.431 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-08-30T19:22:07.787482+0000 57'103 62:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:18:40.201338+0000 0 0 2024-08-30T19:22:22.431 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:22:22.431 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:22:22.431 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:22:22.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-30T19:22:22.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-30T19:22:22.599 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:22:22.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-30T19:22:22.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-30T19:22:22.752 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:22:22.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-30T19:22:22.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-30T19:22:22.909 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-30T19:22:22.923 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-08-30T19:22:22.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-08-30T19:22:22.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-08-30T19:22:23.293 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-30T19:22:23.293 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-08-30T19:22:07.787482+0000 57'103 62:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-30T19:18:58.919309+0000 0'0 2024-08-30T19:18:58.919309+0000 0 0 periodic scrub scheduled @ 2024-09-01T06:18:40.201338+0000 0 0 2024-08-30T19:22:23.293 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-08-30T19:22:23.293 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-30T19:22:23.294 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-08-30T19:22:23.309 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-08-30T19:22:23.310 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-08-30T19:22:23.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:22:23.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-08-30T19:22:23.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:22:23.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-08-30T19:22:23.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:22:23.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-08-30T19:22:23.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-30T19:22:23.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-08-30T19:22:23.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-08-30T19:22:23.523 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-08-30T19:22:23.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-08-30T19:19:40.438209+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-08-30T19:19:40.438209+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-30T19:19:48.675581+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:22:23.529 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-30T19:19:48.675581+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-30T19:19:48.675581+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-30T19:22:23.530 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-08-30T19:19:47.324829+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-30T19:19:48.329998+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-08-30T19:19:47.324829+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-30T19:19:48.329998+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-30T19:19:48.329998+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-30T19:22:23.531 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-08-30T19:21:58.412+0000 7f1cf5f7f640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-08-30T19:22:23.532 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-08-30T19:22:23.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-30T19:22:23.539 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-08-30T19:22:23.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-08-30T19:22:23.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-08-30T19:22:23.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-30T19:22:23.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-30T19:22:24.024 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-08-30T19:22:24.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-08-30T19:22:24.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:22:24.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:22:24.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:22:24.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:22:24.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:22:24.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:22:24.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-30T19:22:24.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:22:24.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:22:24.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:22:24.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:22:24.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:22:24.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:22:24.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:22:24.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:22:24.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:22:24.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:22:24.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:22:24.363 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:22:24.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:22:24.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:22:24.365 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:22:24.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:22:24.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:22:24.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:22:24.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:22:24.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:22:24.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:22:24.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:24.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:22:24.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:22:24.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:22:24.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:22:24.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/divergent-priors 0 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:22:24.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:22:24.396 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:22:24.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:22:24.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:22:24.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:22:24.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:22:24.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:22:24.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:22:24.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:22:24.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:22:24.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-30T19:22:24.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-30T19:22:24.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:22:24.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:24.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27261 2024-08-30T19:22:24.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27261 2024-08-30T19:22:24.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:22:24.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:22:24.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-30T19:22:24.404 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-30T19:22:24.405 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-30T19:22:24.414 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-08-30T19:22:24.415 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-08-30T19:22:24.465 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-30T19:22:24.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ec-error-rollforward 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-08-30T19:22:24.471 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-08-30T19:22:24.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none ' 2024-08-30T19:22:24.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-08-30T19:22:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:22:24.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:22:24.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:22:24.476 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:22:24.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:22:24.476 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:22:24.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:22:24.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:22:24.477 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:22:24.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:22:24.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:22:24.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:22:24.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:22:24.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:22:24.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85101 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:22:24.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:22:24.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-08-30T19:22:24.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:22:24.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:24.483 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:24.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85101 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-08-30T19:22:24.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-08-30T19:22:24.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-08-30T19:22:24.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:22:24.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:22:24.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:22:24.890 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:22:24.890 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:24.890 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:24.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:22:24.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:22:25.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:22:25.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:22:25.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:22:25.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:22:25.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:22:25.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:22:25.178 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:22:25.178 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85101/ceph-mon.a.asok 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:22:25.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85101/ceph-mon.a.asok config get fsid 2024-08-30T19:22:25.296 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:22:25.297 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85101/ceph-mon.a.asok 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:22:25.298 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85101/ceph-mon.a.asok config get mon_host 2024-08-30T19:22:25.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-08-30T19:22:25.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:22:26.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:22:26.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:22:26.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:22:26.315 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:22:26.315 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:26.315 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:26.315 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:22:26.315 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:22:26.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:22:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-08-30T19:22:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-30T19:22:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:22:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:22:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:22:26.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:22:26.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03bd994c-8472-4bcd-9612-b8b1381da5bf 2024-08-30T19:22:26.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 03bd994c-8472-4bcd-9612-b8b1381da5bf' 2024-08-30T19:22:26.351 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 03bd994c-8472-4bcd-9612-b8b1381da5bf 2024-08-30T19:22:26.351 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:22:26.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyG9Jm3bAhFhAAIUDdQknJLPSOH6wd71A4Qg== 2024-08-30T19:22:26.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyG9Jm3bAhFhAAIUDdQknJLPSOH6wd71A4Qg=="}' 2024-08-30T19:22:26.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03bd994c-8472-4bcd-9612-b8b1381da5bf -i td/ec-error-rollforward/0/new.json 2024-08-30T19:22:26.636 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:22:26.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-08-30T19:22:26.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyG9Jm3bAhFhAAIUDdQknJLPSOH6wd71A4Qg== --osd-uuid 03bd994c-8472-4bcd-9612-b8b1381da5bf 2024-08-30T19:22:26.691 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:26.696+0000 7f8bad0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:26.705 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:26.712+0000 7f8bad0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:26.706 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:26.712+0000 7f8bad0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:26.706 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:26.712+0000 7f8bad0f07c0 -1 bdev(0x5595a201ea80 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:22:26.706 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:26.712+0000 7f8bad0f07c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-08-30T19:22:33.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-08-30T19:22:33.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:22:33.105 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:22:33.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:22:33.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:22:33.657 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:22:33.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:22:33.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:22:33.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:22:33.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:22:33.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:22:33.686 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:33.691+0000 7fa96b5c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:33.688 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:33.691+0000 7fa96b5c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:33.689 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:33.695+0000 7fa96b5c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:22:34.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:22:34.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:22:34.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:22:34.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:22:34.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:22:34.041 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:22:34.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:34.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:22:34.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:34.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:34.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:34.756 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:34.759+0000 7fa96b5c27c0 -1 Falling back to public interface 2024-08-30T19:22:35.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:35.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:35.523 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:22:35.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:22:35.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:35.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:35.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:36.926 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:22:36.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:36.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:36.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:22:36.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:36.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:37.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:37.674 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:22:37.679+0000 7fa96b5c27c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:22:38.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:38.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:38.333 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:22:38.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:22:38.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:38.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:38.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:39.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:39.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:39.873 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:22:39.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:22:39.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:39.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:40.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:41.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:41.338 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:22:41.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:41.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:22:41.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:41.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:41.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:42.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:42.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:42.800 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:22:42.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:22:42.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:42.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:43.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:44.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:44.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:44.261 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:22:44.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:22:44.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:44.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:44.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:45.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:45.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:45.876 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:22:45.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:22:45.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:45.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:46.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:47.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:47.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:47.334 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:22:47.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:22:47.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:47.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:47.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:48.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:48.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:48.831 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:22:48.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:22:48.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:48.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:49.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:50.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:50.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:50.282 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:22:50.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:22:50.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:50.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:50.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:51.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:51.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:51.760 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:22:51.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:22:51.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:51.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:52.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:53.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:53.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:53.387 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:22:53.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:22:53.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:53.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:54.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:55.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:55.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:55.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:22:55.242 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:22:55.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:55.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:56.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:57.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:57.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:57.137 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:22:57.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:22:57.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:57.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:57.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:22:58.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:22:58.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:22:58.999 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:22:58.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:22:59.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:22:59.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:22:59.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:00.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:00.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:00.580 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:23:00.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:23:00.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:00.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:23:01.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:02.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:02.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:02.454 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:23:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:23:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:23:03.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:04.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:04.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:04.045 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:23:04.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:23:04.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:04.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:23:04.444 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2145909888,v1:127.0.0.1:6803/2145909888] [v2:127.0.0.1:6804/2145909888,v1:127.0.0.1:6805/2145909888] exists,up 03bd994c-8472-4bcd-9612-b8b1381da5bf 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-08-30T19:23:04.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:23:04.447 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:23:04.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:23:04.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:23:04.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:23:04.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca5cfe4c-95e9-4b49-926d-f83e18553f3c 2024-08-30T19:23:04.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ca5cfe4c-95e9-4b49-926d-f83e18553f3c' 2024-08-30T19:23:04.451 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 ca5cfe4c-95e9-4b49-926d-f83e18553f3c 2024-08-30T19:23:04.451 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:23:04.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYHNJmaz5FHBAAWnQfS+w25/knsRCBvCWuVw== 2024-08-30T19:23:04.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYHNJmaz5FHBAAWnQfS+w25/knsRCBvCWuVw=="}' 2024-08-30T19:23:04.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca5cfe4c-95e9-4b49-926d-f83e18553f3c -i td/ec-error-rollforward/1/new.json 2024-08-30T19:23:04.951 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:23:04.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-08-30T19:23:04.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYHNJmaz5FHBAAWnQfS+w25/knsRCBvCWuVw== --osd-uuid ca5cfe4c-95e9-4b49-926d-f83e18553f3c 2024-08-30T19:23:05.000 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:05.002+0000 7fcb6928a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:05.071 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:05.074+0000 7fcb6928a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:05.073 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:05.074+0000 7fcb6928a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:05.073 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:05.078+0000 7fcb6928a7c0 -1 bdev(0x55f5bac6aa80 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:23:05.073 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:05.078+0000 7fcb6928a7c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-08-30T19:23:11.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-08-30T19:23:11.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:23:11.859 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:23:11.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:23:11.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:23:12.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:23:12.410 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:23:12.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:23:12.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:23:12.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:23:12.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:23:12.441 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:12.442+0000 7fda34a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:12.466 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:12.470+0000 7fda34a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:12.468 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:12.470+0000 7fda34a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:23:12.834 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:23:12.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:12.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:13.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:13.708 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:13.710+0000 7fda34a5d7c0 -1 Falling back to public interface 2024-08-30T19:23:14.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:14.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:14.298 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:23:14.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:23:14.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:14.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:14.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:15.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:15.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:15.719 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:23:15.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:23:15.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:15.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:16.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:16.764 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:16.766+0000 7fda34a5d7c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:23:17.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:17.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:17.129 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:23:17.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:23:17.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:17.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:17.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:18.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:18.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:18.615 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:23:18.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:23:18.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:18.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:19.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:20.138 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:23:20.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:23:20.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:20.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:20.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:21.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:21.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:21.645 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:23:21.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:23:21.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:21.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:22.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:23.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:23.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:23.088 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:23:23.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:23:23.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:23.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:23.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:24.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:24.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:24.681 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:23:24.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:23:24.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:24.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:25.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:26.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:26.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:26.097 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:23:26.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:23:26.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:26.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:26.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:27.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:27.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:27.873 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:23:27.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:23:27.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:27.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:28.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:29.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:29.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:29.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:23:29.336 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:23:29.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:29.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:29.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:30.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:30.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:30.771 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:23:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:23:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:31.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:32.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:32.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:32.228 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:23:32.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:23:32.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:32.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:33.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:34.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:34.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:34.134 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:23:34.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:23:34.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:34.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:34.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:35.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:35.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:35.719 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:23:35.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:23:35.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:35.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:36.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:37.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:37.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:37.553 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:23:37.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:23:37.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:37.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:38.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:39.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:39.354 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:23:39.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:39.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:23:39.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:39.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:39.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:40.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:40.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:40.832 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:23:40.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:23:40.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:40.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:41.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:42.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:42.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:42.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:23:42.580 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:23:42.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:42.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:23:43.009 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1724816658,v1:127.0.0.1:6811/1724816658] [v2:127.0.0.1:6812/1724816658,v1:127.0.0.1:6813/1724816658] exists,up ca5cfe4c-95e9-4b49-926d-f83e18553f3c 2024-08-30T19:23:43.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:23:43.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:23:43.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:23:43.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-08-30T19:23:43.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:23:43.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:23:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:23:43.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-08-30T19:23:43.016 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:23:43.016 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 ee0b687e-1b33-4590-9e47-30bcf22d9ee8 2024-08-30T19:23:43.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee0b687e-1b33-4590-9e47-30bcf22d9ee8 2024-08-30T19:23:43.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ee0b687e-1b33-4590-9e47-30bcf22d9ee8' 2024-08-30T19:23:43.017 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:23:43.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/HNJm0PJfAhAAnoiqdgV1UEPsq8HNU9b0Fw== 2024-08-30T19:23:43.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/HNJm0PJfAhAAnoiqdgV1UEPsq8HNU9b0Fw=="}' 2024-08-30T19:23:43.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee0b687e-1b33-4590-9e47-30bcf22d9ee8 -i td/ec-error-rollforward/2/new.json 2024-08-30T19:23:43.541 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:23:43.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-08-30T19:23:43.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/HNJm0PJfAhAAnoiqdgV1UEPsq8HNU9b0Fw== --osd-uuid ee0b687e-1b33-4590-9e47-30bcf22d9ee8 2024-08-30T19:23:43.591 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:43.593+0000 7fde67dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:43.614 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:43.617+0000 7fde67dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:43.615 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:43.617+0000 7fde67dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:43.615 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:43.617+0000 7fde67dae7c0 -1 bdev(0x55571098ca80 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:23:43.616 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:43.617+0000 7fde67dae7c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-08-30T19:23:50.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-08-30T19:23:50.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:23:50.718 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:23:50.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:23:50.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:23:51.293 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:23:51.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:23:51.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:23:51.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:23:51.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:23:51.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:23:51.323 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:51.325+0000 7fdc81f117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:51.334 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:51.333+0000 7fdc81f117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:51.335 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:51.337+0000 7fdc81f117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:23:51.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:51.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:23:51.701 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:23:51.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:51.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:52.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:52.885 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:52.885+0000 7fdc81f117c0 -1 Falling back to public interface 2024-08-30T19:23:53.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:53.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:53.129 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:23:53.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:23:53.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:53.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:53.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:54.560 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:23:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:23:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:54.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:55.799 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:23:55.801+0000 7fdc81f117c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:23:55.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:55.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:55.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:23:55.966 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:23:55.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:55.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:56.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:57.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:57.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:57.615 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:23:57.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:23:57.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:57.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:58.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:23:59.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:23:59.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:23:59.046 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:23:59.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:23:59.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:23:59.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:23:59.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:00.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:00.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:00.525 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:24:00.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:24:00.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:00.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:01.151 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:02.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:02.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:02.154 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:24:02.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:24:02.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:02.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:02.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:03.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:03.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:03.708 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:24:03.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:24:03.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:03.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:04.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:05.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:05.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:05.174 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:24:05.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:24:05.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:05.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:05.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:06.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:06.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:06.714 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:24:06.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:24:06.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:06.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:07.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:08.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:08.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:08.175 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:24:08.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:24:08.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:08.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:08.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:09.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:09.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:09.724 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:24:09.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:24:09.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:09.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:10.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:11.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:11.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:11.530 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:24:11.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:24:11.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:11.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:11.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:12.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:12.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:12.974 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:24:12.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:24:12.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:12.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:14.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:15.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:15.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:15.342 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:24:15.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:24:15.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:15.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:16.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:17.156 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:24:17.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:17.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:17.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:24:17.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:17.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:17.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:18.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:18.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:18.973 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:24:18.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:24:18.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:18.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:19.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:20.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:20.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:20.968 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:24:20.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:24:20.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:20.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:24:21.451 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/800061136,v1:127.0.0.1:6819/800061136] [v2:127.0.0.1:6820/800061136,v1:127.0.0.1:6821/800061136] exists,up ee0b687e-1b33-4590-9e47-30bcf22d9ee8 2024-08-30T19:24:21.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:24:21.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:24:21.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:24:21.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:24:21.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-08-30T19:24:21.459 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:24:21.459 INFO:tasks.workunit.client.0.smithi079.stdout:add osd3 e657cb47-acf5-4577-97fd-8a48c76434e3 2024-08-30T19:24:21.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e657cb47-acf5-4577-97fd-8a48c76434e3 2024-08-30T19:24:21.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e657cb47-acf5-4577-97fd-8a48c76434e3' 2024-08-30T19:24:21.459 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:24:21.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlHNJmmCJoHBAAahcopuFzGUTwGW8+5wqk+A== 2024-08-30T19:24:21.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlHNJmmCJoHBAAahcopuFzGUTwGW8+5wqk+A=="}' 2024-08-30T19:24:21.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e657cb47-acf5-4577-97fd-8a48c76434e3 -i td/ec-error-rollforward/3/new.json 2024-08-30T19:24:21.974 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:24:21.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-08-30T19:24:21.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlHNJmmCJoHBAAahcopuFzGUTwGW8+5wqk+A== --osd-uuid e657cb47-acf5-4577-97fd-8a48c76434e3 2024-08-30T19:24:22.022 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:22.028+0000 7fdc845d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:22.037 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:22.040+0000 7fdc845d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:22.038 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:22.040+0000 7fdc845d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:22.038 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:22.044+0000 7fdc845d47c0 -1 bdev(0x563051788a80 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-08-30T19:24:22.039 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:22.044+0000 7fdc845d47c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-08-30T19:24:28.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-08-30T19:24:28.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:24:28.877 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd3 key to auth repository 2024-08-30T19:24:28.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-30T19:24:28.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:24:29.433 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:24:29.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-30T19:24:29.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:24:29.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:24:29.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:24:29.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:24:29.462 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:29.468+0000 7f26db21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:29.464 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:29.468+0000 7f26db21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:29.465 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:29.468+0000 7f26db21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:29.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:24:29.896 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:24:29.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:29.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:30.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:30.485 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:30.488+0000 7f26db21b7c0 -1 Falling back to public interface 2024-08-30T19:24:31.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:31.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:31.321 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:24:31.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:24:31.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:31.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:31.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:32.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:32.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:32.737 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:24:32.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:24:32.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:32.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:33.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:33.382 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:24:33.387+0000 7f26db21b7c0 -1 osd.3 0 log_to_monitors true 2024-08-30T19:24:34.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:34.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:34.175 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:24:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:24:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:34.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:35.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:35.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:35.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:24:35.681 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:24:35.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:35.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:36.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:37.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:37.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:37.218 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:24:37.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:24:37.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:37.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:37.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:38.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:38.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:38.802 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:24:38.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:24:38.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:38.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:39.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:40.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:40.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:40.386 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:24:40.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:24:40.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:40.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:40.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:41.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:41.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:41.810 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:24:41.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:24:41.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:41.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:42.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:43.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:43.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:43.415 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:24:43.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:24:43.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:43.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:43.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:44.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:44.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:44.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:24:44.904 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:24:44.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:44.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:45.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:46.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:46.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:46.391 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:24:46.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:24:46.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:46.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:46.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:47.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:47.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:47.836 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:24:47.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:24:47.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:47.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:48.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:49.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:49.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:49.634 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:24:49.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:24:49.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:49.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:50.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:51.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:51.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:51.472 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:52.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:53.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:53.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:53.360 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:24:53.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:24:53.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:53.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:54.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:55.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:55.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:55.186 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:24:55.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:24:55.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:55.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:56.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:57.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:57.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:57.369 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:24:57.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:24:57.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:57.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:57.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:24:58.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:24:58.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:24:58.825 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:24:58.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:24:58.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:24:58.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:24:59.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:00.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:00.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:00.426 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:25:00.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:25:00.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:00.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:25:00.959 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3586278221,v1:127.0.0.1:6827/3586278221] [v2:127.0.0.1:6828/3586278221,v1:127.0.0.1:6829/3586278221] exists,up e657cb47-acf5-4577-97fd-8a48c76434e3 2024-08-30T19:25:00.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:25:00.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:25:00.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:25:00.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-08-30T19:25:01.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-08-30T19:25:02.363 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'ec' already exists 2024-08-30T19:25:02.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-08-30T19:25:04.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-08-30T19:25:04.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 89784 2024-08-30T19:25:04.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=92591 2024-08-30T19:25:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-08-30T19:25:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 92612' 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 92613' 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 92614' 2024-08-30T19:25:05.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-08-30T19:25:05.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-08-30T19:25:06.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-08-30T19:25:06.184 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:25:06.184 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:25:06.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:25:06.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:25:06.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:25:06.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:25:06.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 92591 92612 92613 92614 2024-08-30T19:25:06.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:25:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:25:06.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-08-30T19:25:06.506 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:25:06.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:25:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:06.539+0000 7fc8b3bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:06.543+0000 7fc8b3bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:06.540 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:06.543+0000 7fc8b3bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:06.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:25:06.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:25:06.922 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:06.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:07.344 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2145909888,v1:127.0.0.1:6803/2145909888] [v2:127.0.0.1:6804/2145909888,v1:127.0.0.1:6805/2145909888] exists,up 03bd994c-8472-4bcd-9612-b8b1381da5bf 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:25:07.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:07.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-08-30T19:25:07.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:25:07.351 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:25:07.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:25:07.351 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-08-30T19:25:07.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:25:07.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:25:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:25:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:25:07.376 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:07.378+0000 7f7eb62247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:07.379 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:07.382+0000 7f7eb62247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:07.380 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:07.382+0000 7f7eb62247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:07.540 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:07.542+0000 7fc8b3bc47c0 -1 Falling back to public interface 2024-08-30T19:25:07.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:25:07.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:25:07.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:25:07.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:25:07.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:25:07.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:07.826 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:07.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:25:07.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:07.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:25:08.247 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1724816658,v1:127.0.0.1:6811/1724816658] [v2:127.0.0.1:6812/1724816658,v1:127.0.0.1:6813/1724816658] exists,up ca5cfe4c-95e9-4b49-926d-f83e18553f3c 2024-08-30T19:25:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:25:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:25:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-08-30T19:25:08.252 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:25:08.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:25:08.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:25:08.253 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-08-30T19:25:08.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:25:08.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:25:08.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:25:08.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:25:08.274 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:08.278+0000 7feb7f4b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:08.280 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:08.282+0000 7feb7f4b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:08.281 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:08.282+0000 7feb7f4b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:08.616 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:08.618+0000 7f7eb62247c0 -1 Falling back to public interface 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:25:08.659 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:08.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:08.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:25:08.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:08.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:25:09.077 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/800061136,v1:127.0.0.1:6819/800061136] [v2:127.0.0.1:6820/800061136,v1:127.0.0.1:6821/800061136] exists,up ee0b687e-1b33-4590-9e47-30bcf22d9ee8 2024-08-30T19:25:09.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:25:09.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-08-30T19:25:09.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:25:09.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-30T19:25:09.083 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:25:09.084 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-08-30T19:25:09.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=7ffd566c-7dd0-4838-8d70-d8ccdcfbaf43 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:25:09.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-30T19:25:09.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:25:09.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:25:09.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:25:09.107 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:09.110+0000 7f75460707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:09.142 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:09.146+0000 7f75460707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:09.144 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:09.146+0000 7f75460707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:25:09.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:25:09.477 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:09.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:25:09.583 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:09.586+0000 7feb7f4b57c0 -1 Falling back to public interface 2024-08-30T19:25:09.921 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3586278221,v1:127.0.0.1:6827/3586278221] [v2:127.0.0.1:6828/3586278221,v1:127.0.0.1:6829/3586278221] exists,up e657cb47-acf5-4577-97fd-8a48c76434e3 2024-08-30T19:25:09.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:25:09.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:25:09.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:25:10.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:25:10.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:25:10.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:25:10.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:25:10.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:25:10.008 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:25:10.428 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:10.430+0000 7f75460707c0 -1 Falling back to public interface 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:25:10.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:25:11.179 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:11.182+0000 7fc8b3bc47c0 -1 osd.0 25 log_to_monitors true 2024-08-30T19:25:13.116 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:13.118+0000 7f7eb62247c0 -1 osd.1 25 log_to_monitors true 2024-08-30T19:25:14.365 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:14.366+0000 7feb7f4b57c0 -1 osd.2 24 log_to_monitors true 2024-08-30T19:25:14.833 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:14.834+0000 7f75460707c0 -1 osd.3 25 log_to_monitors true 2024-08-30T19:25:15.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084291 2024-08-30T19:25:15.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084291 2024-08-30T19:25:15.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291' 2024-08-30T19:25:15.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:25:15.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:25:15.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986179 2024-08-30T19:25:15.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986179 2024-08-30T19:25:15.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2024-08-30T19:25:15.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:25:15.827 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:25:15.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822659 2024-08-30T19:25:15.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822659 2024-08-30T19:25:15.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-154618822659' 2024-08-30T19:25:15.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:25:15.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-30T19:25:16.168 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-30T19:25:16.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-08-30T19:25:16.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-08-30T19:25:16.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-08-30T19:25:16.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:25:16.171 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084291 2024-08-30T19:25:16.171 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:25:16.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:25:16.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084291 2024-08-30T19:25:16.173 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:25:16.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084291 2024-08-30T19:25:16.174 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 120259084291 2024-08-30T19:25:16.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2024-08-30T19:25:16.175 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:25:16.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084291 2024-08-30T19:25:16.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:25:17.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:25:17.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:25:18.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084292 -lt 120259084291 2024-08-30T19:25:18.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:25:18.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-133143986179 2024-08-30T19:25:18.218 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:25:18.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:25:18.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-133143986179 2024-08-30T19:25:18.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:25:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986179 2024-08-30T19:25:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2024-08-30T19:25:18.222 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 133143986179 2024-08-30T19:25:18.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:25:18.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986179 -lt 133143986179 2024-08-30T19:25:18.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:25:18.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:25:18.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822659 2024-08-30T19:25:18.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:25:18.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822659 2024-08-30T19:25:18.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:25:18.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822659 2024-08-30T19:25:18.694 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 154618822659 2024-08-30T19:25:18.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-08-30T19:25:18.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:25:19.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822659 -lt 154618822659 2024-08-30T19:25:19.249 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:25:19.249 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:25:19.249 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:25:19.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-30T19:25:19.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:25:19.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:25:19.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:25:19.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:25:19.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:25:19.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:25:19.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:25:20.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-30T19:25:20.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:25:20.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:25:20.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-30T19:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-08-30T19:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-30T19:25:20.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:25:20.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-30T19:25:20.781 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:25:20.781 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:25:20.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:25:20.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:25:20.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:25:20.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:25:20.892 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:25:20.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:25:20.892 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:25:20.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:25:20.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:25:20.893 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:25:20.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:20.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:25:20.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:25:20.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:20.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:25:20.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:25:20.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-30T19:25:20.915 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:25:20.915 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:20.915 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:20.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85101 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ec-error-rollforward 0 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:25:20.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:25:20.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:25:20.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:25:20.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:25:20.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:25:20.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:25:20.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:25:20.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:25:20.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:20.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:25:20.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:25:20.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:20.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:25:20.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-30T19:25:20.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-30T19:25:20.924 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:25:20.924 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:20.924 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85101 2024-08-30T19:25:20.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85101 2024-08-30T19:25:20.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:25:20.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:25:20.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-30T19:25:20.927 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-30T19:25:20.927 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-30T19:25:20.979 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-08-30T19:25:20.980 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-08-30T19:25:21.031 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-30T19:25:21.036 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-30T19:25:21.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-bench 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-08-30T19:25:21.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-08-30T19:25:21.038 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-08-30T19:25:21.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=b9c10c8f-2122-4999-a9ba-5148a6c53591 --auth-supported=none ' 2024-08-30T19:25:21.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-08-30T19:25:21.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-08-30T19:25:21.039 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-08-30T19:25:21.039 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-30T19:25:21.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-08-30T19:25:21.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-08-30T19:25:21.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-08-30T19:25:21.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-08-30T19:25:21.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-08-30T19:25:21.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-30T19:25:21.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:25:21.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-30T19:25:21.042 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:25:21.042 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:25:21.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:25:21.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:25:21.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:25:21.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:25:21.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:21.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:25:21.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:25:21.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95120 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:25:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-08-30T19:25:21.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:25:21.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:21.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:21.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95120 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-08-30T19:25:21.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:25:21.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:25:21.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:25:21.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-08-30T19:25:21.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-08-30T19:25:21.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:25:21.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:21.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:21.479 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:21.479 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:21.479 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:21.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95120/$cluster-$name.asok' 2024-08-30T19:25:21.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.95120/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:25:21.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:25:21.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:25:21.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:25:21.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:25:21.693 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:25:21.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:25:21.694 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:25:21.694 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:25:21.694 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95120/ceph-mon.a.asok 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:25:21.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95120/ceph-mon.a.asok config get fsid 2024-08-30T19:25:21.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:25:21.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:25:21.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:25:21.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:25:21.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:25:21.808 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:25:21.808 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:25:21.808 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:25:21.808 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:25:21.808 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:21.809 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:21.809 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95120/ceph-mon.a.asok 2024-08-30T19:25:21.809 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:25:21.809 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95120/ceph-mon.a.asok config get mon_host 2024-08-30T19:25:21.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-08-30T19:25:21.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:25:22.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:25:22.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:22.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:22.819 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:22.820 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:22.820 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:22.820 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95120/$cluster-$name.asok' 2024-08-30T19:25:22.820 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:25:22.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.95120/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:25:22.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-08-30T19:25:22.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-08-30T19:25:22.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b9c10c8f-2122-4999-a9ba-5148a6c53591 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:25:22.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95120/$cluster-$name.asok' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95120/$cluster-$name.asok' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:25:22.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd5a50ef-c814-4283-8b18-90ac5d4e5165 2024-08-30T19:25:22.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dd5a50ef-c814-4283-8b18-90ac5d4e5165' 2024-08-30T19:25:22.896 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 dd5a50ef-c814-4283-8b18-90ac5d4e5165 2024-08-30T19:25:22.896 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:25:22.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiHNJmKyFyNhAAhdDo2hjOWYhTcNg6wMibOA== 2024-08-30T19:25:22.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiHNJmKyFyNhAAhdDo2hjOWYhTcNg6wMibOA=="}' 2024-08-30T19:25:22.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd5a50ef-c814-4283-8b18-90ac5d4e5165 -i td/osd-bench/0/new.json 2024-08-30T19:25:23.218 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:23.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-08-30T19:25:23.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b9c10c8f-2122-4999-a9ba-5148a6c53591 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.95120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiHNJmKyFyNhAAhdDo2hjOWYhTcNg6wMibOA== --osd-uuid dd5a50ef-c814-4283-8b18-90ac5d4e5165 2024-08-30T19:25:23.266 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:23.266+0000 7f49714ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:23.296 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:23.298+0000 7f49714ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:23.296 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:23.298+0000 7f49714ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:23.297 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:23.298+0000 7f49714ee7c0 -1 bdev(0x5624fe814a80 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:25:23.297 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:23.298+0000 7f49714ee7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-08-30T19:25:29.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-08-30T19:25:29.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:25:29.576 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:25:29.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:25:29.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:25:30.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:25:30.142 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:25:30.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:25:30.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b9c10c8f-2122-4999-a9ba-5148a6c53591 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.95120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:25:30.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:25:30.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:25:30.172 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:30.174+0000 7fccad5427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:30.174 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:30.174+0000 7fccad5427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:30.175 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:30.178+0000 7fccad5427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:25:30.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:30.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:25:30.590 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:25:30.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:30.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:31.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:31.189 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:31.190+0000 7fccad5427c0 -1 Falling back to public interface 2024-08-30T19:25:32.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:32.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:32.004 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:25:32.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:25:32.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:32.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:32.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:33.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:33.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:33.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:25:33.444 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:25:33.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:33.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:33.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:34.108 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:25:34.102+0000 7fccad5427c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:25:34.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:34.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:34.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:25:34.889 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:25:34.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:34.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:35.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:36.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:36.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:36.311 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:25:36.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:25:36.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:36.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:36.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:37.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:37.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:37.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:25:37.762 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:25:37.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:37.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:38.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:39.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:39.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:39.241 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:25:39.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:25:39.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:39.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:39.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:40.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:40.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:40.687 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:25:40.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:25:40.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:40.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:41.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:42.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:42.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:42.160 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:25:42.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:25:42.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:42.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:42.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:43.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:43.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:43.611 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:25:43.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:25:43.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:43.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:44.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:45.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:45.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:45.173 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:25:45.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:25:45.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:45.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:45.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:46.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:46.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:46.592 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:25:46.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:25:46.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:46.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:47.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:48.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:48.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:48.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:25:48.149 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:25:48.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:48.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:48.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:49.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:49.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:49.830 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:25:49.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:25:49.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:49.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:51.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:52.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:52.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:52.402 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:25:52.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:25:52.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:52.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:53.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:54.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:54.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:54.085 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:25:54.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:25:54.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:54.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:55.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:56.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:56.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:56.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:25:56.222 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:25:56.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:56.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:57.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:25:58.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:25:58.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:25:58.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:25:58.129 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:25:58.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:25:58.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:25:59.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:26:00.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:26:00.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:26:00.425 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:26:00.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:26:00.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:26:00.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:26:00.842 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3887201296,v1:127.0.0.1:6803/3887201296] [v2:127.0.0.1:6804/3887201296,v1:127.0.0.1:6805/3887201296] exists,up dd5a50ef-c814-4283-8b18-90ac5d4e5165 2024-08-30T19:26:00.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:26:00.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:26:00.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:26:00.843 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-08-30T19:26:00.843 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-08-30T19:26:00.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-08-30T19:26:00.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-08-30T19:26:00.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-08-30T19:26:00.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-08-30T19:26:00.873 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-08-30T19:26:00.874 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-08-30T19:26:00.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-08-30T19:26:00.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-08-30T19:26:00.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=osd_bench_max_block_size 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-08-30T19:26:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-08-30T19:26:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-08-30T19:26:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-08-30T19:26:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-08-30T19:26:01.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=bench_small_size_max_iops 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-08-30T19:26:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-08-30T19:26:01.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-08-30T19:26:01.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-08-30T19:26:01.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-08-30T19:26:01.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-08-30T19:26:01.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-08-30T19:26:01.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-08-30T19:26:01.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-08-30T19:26:01.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-08-30T19:26:01.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-08-30T19:26:01.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-08-30T19:26:01.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-08-30T19:26:01.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-08-30T19:26:59.156 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:26:59.156 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 1073741824, 2024-08-30T19:26:59.157 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 4194304, 2024-08-30T19:26:59.157 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 27.795335262000002, 2024-08-30T19:26:59.157 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 38630288.639401697, 2024-08-30T19:26:59.157 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 9.2101785276893846 2024-08-30T19:26:59.157 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:26:59.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 10, 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 14456, 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 0.041791552000000003, 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 239.28281007606512, 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 0.01655249101245608 2024-08-30T19:26:59.690 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:26:59.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-08-30T19:27:00.032 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-08-30T19:27:00.032 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 1, 2024-08-30T19:27:00.032 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 14456, 2024-08-30T19:27:00.033 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 0.047573727000000003, 2024-08-30T19:27:00.033 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 21.020005432830605, 2024-08-30T19:27:00.033 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 0.0014540678910369815 2024-08-30T19:27:00.033 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-08-30T19:27:00.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-08-30T19:27:00.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-30T19:27:00.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:27:00.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-30T19:27:00.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:27:00.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:27:00.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:27:00.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:27:00.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:27:00.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:27:00.371 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:27:00.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:27:00.373 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:27:00.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:27:00.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:27:00.375 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:27:00.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:27:00.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:27:00.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:27:00.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:27:00.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-30T19:27:00.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:27:00.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:00.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:27:00.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95120 2024-08-30T19:27:00.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:27:00.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:27:00.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-bench 0 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:27:00.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:27:00.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:27:00.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:27:00.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:27:00.415 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:27:00.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:27:00.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:27:00.416 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:27:00.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:27:00.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:27:00.418 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:27:00.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-30T19:27:00.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-30T19:27:00.421 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:27:00.421 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:00.421 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95120 2024-08-30T19:27:00.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95120 2024-08-30T19:27:00.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:27:00.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:27:00.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-08-30T19:27:00.424 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-30T19:27:00.424 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-30T19:27:00.479 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-08-30T19:27:00.480 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-08-30T19:27:00.534 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-30T19:27:00.542 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-30T19:27:00.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-08-30T19:27:00.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-08-30T19:27:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-08-30T19:27:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-bluefs-volume-ops 2024-08-30T19:27:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-08-30T19:27:00.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-30T19:27:00.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_bluestore2 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_bluestore_expand' 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:27:00.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:27:00.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:27:00.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:27:00.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:27:00.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:27:00.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:27:00.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:27:00.549 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:27:00.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:27:00.551 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:27:00.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:27:00.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:27:00.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:27:00.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:27:00.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:27:00.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:27:00.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:27:00.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:27:00.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:27:00.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:27:00.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:00.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:00.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:27:00.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:27:00.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:27:00.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-30T19:27:00.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:27:00.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:00.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:00.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97333 2024-08-30T19:27:00.562 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-08-30T19:27:00.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-08-30T19:27:00.564 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:27:00.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-30T19:27:00.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:00.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:00.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:27:01.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:27:01.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:27:01.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:27:01.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:27:01.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:27:01.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:01.175 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:01.176 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:27:01.176 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:27:01.176 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get fsid 2024-08-30T19:27:01.291 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:27:01.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:27:01.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:27:01.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:27:01.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:27:01.293 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:27:01.293 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:27:01.293 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:27:01.293 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:27:01.293 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:01.294 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:01.294 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:27:01.294 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:27:01.294 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get mon_host 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-30T19:27:01.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:27:02.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:27:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:27:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:27:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:27:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:27:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:27:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:27:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:27:02.351 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:27:02.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66' 2024-08-30T19:27:02.351 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:27:02.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGHdJmewsOFhAAoEfVFJM8gmpDzEujenu3NA== 2024-08-30T19:27:02.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGHdJmewsOFhAAoEfVFJM8gmpDzEujenu3NA=="}' 2024-08-30T19:27:02.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 -i td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:27:02.649 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:27:02.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:27:02.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGHdJmewsOFhAAoEfVFJM8gmpDzEujenu3NA== --osd-uuid bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:27:02.697 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:02.700+0000 7f5dbaa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:02.717 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:02.720+0000 7f5dbaa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:02.718 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:02.724+0000 7f5dbaa597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:02.719 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:02.724+0000 7f5dbaa597c0 -1 bdev(0x56324d7f4a80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:27:02.719 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:02.724+0000 7f5dbaa597c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-30T19:27:12.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-30T19:27:12.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:27:12.095 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:27:12.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:27:12.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:27:12.637 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:27:12.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:27:12.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:27:12.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:27:12.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:27:12.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:27:12.662 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:12.664+0000 7fc4d68ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:12.677 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:12.680+0000 7fc4d68ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:12.678 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:12.684+0000 7fc4d68ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:13.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:27:13.054 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:27:13.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:13.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:13.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:14.437 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:14.439+0000 7fc4d68ae7c0 -1 Falling back to public interface 2024-08-30T19:27:14.524 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:27:14.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:14.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:14.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:27:14.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:14.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:14.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:15.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:15.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:15.929 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:27:15.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:27:15.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:15.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:16.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:17.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:17.342 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:27:17.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:17.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:27:17.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:17.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:17.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:18.742 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:27:18.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:18.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:18.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:27:18.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:18.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:19.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:19.255 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:19.259+0000 7fc4d68ae7c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:27:20.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:20.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:20.148 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:27:20.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:27:20.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:20.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:20.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:21.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:21.570 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:27:21.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:27:21.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:21.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:22.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:23.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:23.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:23.135 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:27:23.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:27:23.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:23.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:23.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:24.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:24.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:24.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:27:24.538 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:27:24.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:24.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:25.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:26.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:26.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:26.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:27:26.014 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:27:26.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:26.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:26.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:27.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:27.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:27.451 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:27:27.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:27:27.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:27.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:28.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:29.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:29.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:29.098 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:27:29.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:27:29.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:29.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:29.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:30.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:30.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:30.527 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:27:30.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:27:30.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:30.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:31.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:32.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:32.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:32.085 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:27:32.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:27:32.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:32.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:32.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:33.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:33.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:33.806 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:27:33.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:27:33.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:33.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:34.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:35.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:35.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:35.663 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:27:35.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:27:35.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:35.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:36.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:37.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:37.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:37.579 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:27:37.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:27:37.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:37.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:38.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:39.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:39.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:39.359 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:27:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:27:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:40.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:41.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:41.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:41.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:27:41.090 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:27:41.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:41.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:41.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:42.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:42.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:42.532 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:27:42.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:27:42.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:42.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:27:42.997 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2844576503,v1:127.0.0.1:6803/2844576503] [v2:127.0.0.1:6804/2844576503,v1:127.0.0.1:6805/2844576503] exists,up bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:27:42.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:27:42.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:27:42.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:27:42.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=98799 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:27:43.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:27:43.001 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:43.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:27:43.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-30T19:27:43.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:27:43.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:27:43.004 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:27:43.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9bbb26fc-0d11-4ab6-8181-d22788708029' 2024-08-30T19:27:43.005 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:27:43.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvHdJmfEyJARAA97PiQ8erdH4BUwWhR7werw== 2024-08-30T19:27:43.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvHdJmfEyJARAA97PiQ8erdH4BUwWhR7werw=="}' 2024-08-30T19:27:43.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9bbb26fc-0d11-4ab6-8181-d22788708029 -i td/osd-bluefs-volume-ops/1/new.json 2024-08-30T19:27:43.484 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:27:43.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-08-30T19:27:43.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvHdJmfEyJARAA97PiQ8erdH4BUwWhR7werw== --osd-uuid 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:27:43.533 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:43.535+0000 7f1c5e7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:43.536 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:43.539+0000 7f1c5e7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:43.537 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:43.539+0000 7f1c5e7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:43.538 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:43.539+0000 7f1c5e7067c0 -1 bdev(0x55f20b906a80 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-08-30T19:27:43.538 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:43.539+0000 7f1c5e7067c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-08-30T19:27:53.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-08-30T19:27:53.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:27:53.473 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-08-30T19:27:53.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-30T19:27:53.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:27:54.029 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:27:54.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-30T19:27:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:27:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:27:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:27:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:27:54.059 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:54.062+0000 7f27c7e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:54.090 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:54.090+0000 7f27c7e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:54.091 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:54.094+0000 7f27c7e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:27:54.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-30T19:27:54.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:27:54.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:27:54.460 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:54.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:27:54.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:55.653 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:27:55.654+0000 7f27c7e567c0 -1 Falling back to public interface 2024-08-30T19:27:55.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:55.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:55.874 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:27:55.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:27:55.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:55.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:27:56.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:57.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:57.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:57.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:27:57.313 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:27:57.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:57.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:27:57.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:27:58.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:27:58.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:27:58.727 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:27:58.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:27:58.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:27:58.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:27:59.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:00.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:00.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:00.180 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:28:00.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:28:00.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:00.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:00.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:01.240 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:01.242+0000 7f27c7e567c0 -1 osd.1 0 log_to_monitors true 2024-08-30T19:28:01.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:01.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:01.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:28:01.590 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:28:01.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:01.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:01.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:02.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:02.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:02.993 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:28:02.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:28:02.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:02.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:03.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:04.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:04.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:04.419 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:28:04.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:28:04.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:04.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:04.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:05.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:05.829 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:28:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:28:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:05.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:06.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:07.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:07.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:07.287 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:28:07.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:28:07.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:07.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:07.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:08.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:08.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:08.754 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:28:08.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:28:08.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:08.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:09.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:10.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:10.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:10.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:28:10.270 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:28:10.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:10.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:10.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:11.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:11.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:11.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:28:11.718 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:28:11.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:11.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:12.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:13.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:13.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:13.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:28:13.169 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:28:13.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:13.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:14.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:15.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:15.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:15.169 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:28:15.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:28:15.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:15.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:16.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:17.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:17.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:17.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:28:17.032 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:28:17.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:17.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:18.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:28:19.049 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:28:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:20.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:21.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:21.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:21.156 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:28:21.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:28:21.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:21.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:22.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:23.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:23.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:23.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:28:23.218 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:28:23.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:23.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:28:23.843 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1766952196,v1:127.0.0.1:6811/1766952196] [v2:127.0.0.1:6812/1766952196,v1:127.0.0.1:6813/1766952196] exists,up 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:28:23.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:28:23.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:28:23.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:28:23.844 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=101355 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:28:23.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:28:23.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:28:23.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-30T19:28:23.851 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:28:23.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:28:23.851 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:28:23.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ae53fa2b-f9b9-48d3-a19a-4f3cdea04088' 2024-08-30T19:28:23.852 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:28:23.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXHdJmlRtLNBAAu+I+KTpDCZ6B3VrfToIMNA== 2024-08-30T19:28:23.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXHdJmlRtLNBAAu+I+KTpDCZ6B3VrfToIMNA=="}' 2024-08-30T19:28:23.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 -i td/osd-bluefs-volume-ops/2/new.json 2024-08-30T19:28:24.521 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:28:24.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-08-30T19:28:24.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXHdJmlRtLNBAAu+I+KTpDCZ6B3VrfToIMNA== --osd-uuid ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:28:24.577 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:24.577+0000 7ff7f49a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:24.579 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:24.581+0000 7ff7f49a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:24.581 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:24.581+0000 7ff7f49a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:24.581 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:24.581+0000 7ff7f49a67c0 -1 bdev(0x55c4ade52a80 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-08-30T19:28:24.581 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:24.581+0000 7ff7f49a67c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-08-30T19:28:34.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-08-30T19:28:34.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:28:34.784 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-08-30T19:28:34.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-30T19:28:34.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:28:35.358 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:28:35.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-30T19:28:35.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:28:35.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:28:35.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:28:35.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:28:35.390 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:35.389+0000 7fb69f1c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:35.415 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:35.413+0000 7fb69f1c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:35.416 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:35.417+0000 7fb69f1c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:28:35.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:35.772 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:28:35.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:28:35.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:35.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:36.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:37.190 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:37.189+0000 7fb69f1c07c0 -1 Falling back to public interface 2024-08-30T19:28:37.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:37.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:37.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:28:37.205 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:28:37.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:37.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:37.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:38.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:38.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:38.689 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:28:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:28:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:39.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:40.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:40.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:40.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:28:40.104 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:28:40.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:40.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:40.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:41.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:41.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:41.506 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:28:41.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:28:41.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:41.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:41.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:42.467 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:28:42.465+0000 7fb69f1c07c0 -1 osd.2 0 log_to_monitors true 2024-08-30T19:28:42.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:42.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:42.963 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:28:42.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:28:42.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:42.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:43.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:44.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:44.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:44.544 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:28:44.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:28:44.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:44.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:45.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:45.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:45.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:28:45.996 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:28:45.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:45.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:46.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:47.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:47.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:47.596 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:28:47.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:28:47.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:47.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:48.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:49.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:49.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:49.114 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:28:49.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:28:49.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:49.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:49.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:50.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:50.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:50.697 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:28:50.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:28:50.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:50.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:51.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:52.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:52.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:52.125 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:28:52.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:28:52.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:52.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:52.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:53.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:53.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:53.600 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:28:53.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:28:53.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:53.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:54.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:55.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:55.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:55.111 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:28:55.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:28:55.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:55.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:56.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:57.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:57.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:57.005 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:28:57.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:28:57.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:57.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:57.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:28:58.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:28:58.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:28:58.970 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:28:58.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:28:58.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:28:58.971 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:28:59.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:00.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:00.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:00.517 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:29:00.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:29:00.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:00.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:29:01.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:02.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:02.262 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:29:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:29:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:29:03.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:04.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:04.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:04.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:29:04.040 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:29:04.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:04.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:29:04.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:05.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:05.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:05.817 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-08-30T19:29:05.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-30T19:29:05.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:05.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:29:06.289 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3553517476,v1:127.0.0.1:6819/3553517476] [v2:127.0.0.1:6820/3553517476,v1:127.0.0.1:6821/3553517476] exists,up ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:29:06.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:29:06.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:29:06.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:29:06.290 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=103884 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:29:06.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:29:06.295 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:29:06.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:29:06.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:29:06.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-30T19:29:06.298 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:29:06.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:29:06.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 fd702c6c-ce07-4321-8699-4c23c2bd6194' 2024-08-30T19:29:06.299 INFO:tasks.workunit.client.0.smithi079.stdout:add osd3 fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:29:06.299 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:29:06.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCHdJmNt3rEhAAIPxbmVgdpPbq7m5Gebnh4g== 2024-08-30T19:29:06.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCHdJmNt3rEhAAIPxbmVgdpPbq7m5Gebnh4g=="}' 2024-08-30T19:29:06.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd702c6c-ce07-4321-8699-4c23c2bd6194 -i td/osd-bluefs-volume-ops/3/new.json 2024-08-30T19:29:06.925 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:29:06.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-08-30T19:29:06.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCHdJmNt3rEhAAIPxbmVgdpPbq7m5Gebnh4g== --osd-uuid fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:29:06.976 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:06.980+0000 7f584bf177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:07.000+0000 7f584bf177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:07.000 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:07.004+0000 7f584bf177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:07.001 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:07.004+0000 7f584bf177c0 -1 bdev(0x55c62a2bea80 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-08-30T19:29:07.001 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:07.004+0000 7f584bf177c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-08-30T19:29:16.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-08-30T19:29:16.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:29:16.690 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd3 key to auth repository 2024-08-30T19:29:16.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-30T19:29:16.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:29:17.238 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:29:17.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-30T19:29:17.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:29:17.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:29:17.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:29:17.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:29:17.269 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:17.272+0000 7f0a7eb987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:17.271 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:17.272+0000 7f0a7eb987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:17.273 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:17.276+0000 7f0a7eb987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:29:17.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-30T19:29:17.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:29:17.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:29:17.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:17.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:18.815 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:18.816+0000 7f0a7eb987c0 -1 Falling back to public interface 2024-08-30T19:29:19.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:19.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:19.118 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:29:19.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:29:19.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:19.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:19.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:20.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:20.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:20.536 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:29:20.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:29:20.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:20.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:20.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:21.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:21.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:21.940 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:29:21.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:29:21.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:21.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:22.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:23.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:23.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:23.381 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:29:23.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:29:23.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:23.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:23.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:23.926 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:29:23.927+0000 7f0a7eb987c0 -1 osd.3 0 log_to_monitors true 2024-08-30T19:29:24.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:24.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:24.891 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:29:24.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:29:24.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:24.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:25.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:26.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:26.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:26.338 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:29:26.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:29:26.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:26.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:26.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:27.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:27.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:27.828 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:29:27.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:29:27.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:27.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:28.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:29.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:29.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:29.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-30T19:29:29.247 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-08-30T19:29:29.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:29.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:29.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:30.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:30.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:30.772 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-08-30T19:29:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-30T19:29:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:30.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:31.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:32.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:32.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:32.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-30T19:29:32.296 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-08-30T19:29:32.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:32.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:32.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:33.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:33.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:33.741 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-08-30T19:29:33.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-30T19:29:33.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:33.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:34.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:35.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:35.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:35.184 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-08-30T19:29:35.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-30T19:29:35.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:35.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:35.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:36.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:36.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:36.640 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-08-30T19:29:36.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-30T19:29:36.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:36.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:37.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:38.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:38.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:38.330 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-08-30T19:29:38.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-30T19:29:38.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:38.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:39.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:40.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:40.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:40.166 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-08-30T19:29:40.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-30T19:29:40.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:40.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:40.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:41.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:41.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:41.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-30T19:29:41.906 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-08-30T19:29:41.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:41.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:42.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:43.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:43.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:43.966 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-08-30T19:29:43.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-30T19:29:43.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:43.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:44.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:29:45.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:29:45.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:29:45.943 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-08-30T19:29:45.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-30T19:29:45.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:29:45.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:29:46.432 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1333122645,v1:127.0.0.1:6827/1333122645] [v2:127.0.0.1:6828/1333122645,v1:127.0.0.1:6829/1333122645] exists,up fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:29:46.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:29:46.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:29:46.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:29:46.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-30T19:29:46.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=106438 2024-08-30T19:29:46.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-08-30T19:29:51.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-08-30T19:29:51.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-30T19:29:52.502 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-08-30T19:29:52.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:29:53.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-30T19:29:53.564 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:29:53.565 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-30T19:29:53.565 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_107778 2024-08-30T19:29:53.565 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:29:53.565 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:29:54.565 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 65 49 0.191387 0.191406 0.228182 0.299491 2024-08-30T19:29:55.565 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 114 98 0.191381 0.191406 0.212393 0.296628 2024-08-30T19:29:56.565 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 178 162 0.210906 0.25 0.243896 0.277189 2024-08-30T19:29:57.565 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 242 226 0.220671 0.25 0.255242 0.271436 2024-08-30T19:29:58.565 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 306 290 0.226529 0.25 0.256231 0.269485 2024-08-30T19:29:59.565 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 368 352 0.229133 0.242188 0.272409 0.268939 2024-08-30T19:30:00.566 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 432 416 0.23211 0.25 0.255145 0.267277 2024-08-30T19:30:01.566 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 492 476 0.232389 0.234375 0.259741 0.264595 2024-08-30T19:30:02.566 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 560 544 0.236078 0.265625 0.273942 0.262268 2024-08-30T19:30:03.566 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 624 608 0.237467 0.25 0.283428 0.261838 2024-08-30T19:30:04.566 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 688 672 0.238603 0.25 0.243769 0.260597 2024-08-30T19:30:05.566 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 752 736 0.23955 0.25 0.313146 0.259583 2024-08-30T19:30:06.566 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 816 800 0.240351 0.25 0.247308 0.258417 2024-08-30T19:30:07.567 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 880 864 0.241039 0.25 0.243744 0.258133 2024-08-30T19:30:08.567 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 937 921 0.239812 0.222656 0.251477 0.258115 2024-08-30T19:30:09.567 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 1007 991 0.241911 0.273438 0.243544 0.256928 2024-08-30T19:30:10.567 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1072 1056 0.242615 0.253906 0.223136 0.255938 2024-08-30T19:30:11.567 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1136 1120 0.243023 0.25 0.242203 0.256094 2024-08-30T19:30:12.567 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1189 1173 0.241127 0.207031 0.267565 0.257484 2024-08-30T19:30:13.567 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:30:13.571838+0000 min lat: 0.145046 max lat: 0.459436 avg lat: 0.258969 2024-08-30T19:30:13.568 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:30:13.568 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1248 1232 0.240592 0.230469 0.254747 0.258969 2024-08-30T19:30:14.568 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1312 1296 0.241038 0.25 0.240584 0.25802 2024-08-30T19:30:15.568 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1376 1360 0.241445 0.25 0.276724 0.257367 2024-08-30T19:30:16.568 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1440 1424 0.241815 0.25 0.231834 0.257485 2024-08-30T19:30:17.568 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1495 1479 0.24069 0.214844 0.277273 0.257589 2024-08-30T19:30:18.568 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1568 1552 0.242467 0.285156 0.242266 0.257114 2024-08-30T19:30:19.568 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1632 1616 0.242756 0.25 0.222546 0.256331 2024-08-30T19:30:20.568 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1702 1686 0.243891 0.273438 0.218136 0.255232 2024-08-30T19:30:21.569 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1766 1750 0.244108 0.25 0.269007 0.254597 2024-08-30T19:30:22.569 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1831 1815 0.244445 0.253906 0.266698 0.254355 2024-08-30T19:30:23.569 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 1857 1841 0.239682 0.101562 0.451687 0.254484 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.3461 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 1857 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.23904 2024-08-30T19:30:24.569 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0336573 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.285156 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.101562 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 61 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 8.61628 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 73 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 26 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.260357 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0710561 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.97402 2024-08-30T19:30:24.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.145046 2024-08-30T19:30:24.703 INFO:tasks.workunit.client.0.smithi079.stdout:after bench 2024-08-30T19:30:24.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-08-30T19:30:24.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98799 2024-08-30T19:30:24.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-08-30T19:30:25.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98799 2024-08-30T19:30:25.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (98799) - No such process 2024-08-30T19:30:25.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-08-30T19:30:26.306 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:30:26.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101355 2024-08-30T19:30:26.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-08-30T19:30:27.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101355 2024-08-30T19:30:27.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (101355) - No such process 2024-08-30T19:30:27.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-08-30T19:30:27.863 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:30:27.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103884 2024-08-30T19:30:27.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-08-30T19:30:28.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103884 2024-08-30T19:30:28.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (103884) - No such process 2024-08-30T19:30:28.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-08-30T19:30:29.338 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:30:29.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106438 2024-08-30T19:30:29.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-08-30T19:30:30.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106438 2024-08-30T19:30:30.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (106438) - No such process 2024-08-30T19:30:30.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-08-30T19:30:30.899 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-08-30T19:30:30.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:30:33.280 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:30:33.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:30:35.676 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:30:35.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:30:38.068 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:30:38.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:30:40.708 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:30:40.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-08-30T19:30:40.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-08-30T19:30:40.779 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:30:41.964 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-30T19:30:41.964 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-30T19:30:41.964 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100000000 : using 0x805b8000(2.0 GiB) 2024-08-30T19:30:41.964 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-08-30T19:30:41.964 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-08-30T19:30:41.992 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-08-30T19:30:41.992 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-08-30T19:30:42.001 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-08-30T19:30:42.001 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-08-30T19:30:42.134 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4294967296 2024-08-30T19:30:47.582 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-08-30T19:30:47.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-08-30T19:30:47.610 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:30:48.798 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-30T19:30:48.798 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-30T19:30:48.798 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x101000000 : using 0x815a7000(2.0 GiB) 2024-08-30T19:30:48.798 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-08-30T19:30:48.798 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-08-30T19:30:48.827 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-08-30T19:30:48.827 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-08-30T19:30:48.837 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-08-30T19:30:48.838 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-08-30T19:30:49.004 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4311744512 2024-08-30T19:30:54.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-08-30T19:30:54.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-08-30T19:30:54.638 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:30:55.854 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-30T19:30:55.855 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-30T19:30:55.855 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100020000 : using 0x805d0000(2.0 GiB) 2024-08-30T19:30:55.855 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-08-30T19:30:55.855 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-08-30T19:30:55.863 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-08-30T19:30:55.863 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-08-30T19:30:55.871 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-08-30T19:30:55.871 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-08-30T19:30:56.005 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4295098368 2024-08-30T19:31:01.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-08-30T19:31:01.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-08-30T19:31:01.465 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:02.673 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-30T19:31:02.673 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-30T19:31:02.673 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0xfff00000 : using 0x804bd000(2.0 GiB) 2024-08-30T19:31:02.673 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-08-30T19:31:02.673 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-08-30T19:31:02.687 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-08-30T19:31:02.687 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-08-30T19:31:02.695 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-08-30T19:31:02.695 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-08-30T19:31:02.804 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4293918720 2024-08-30T19:31:08.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:31:10.504 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:10.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:31:12.912 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:12.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:31:15.337 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:15.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:31:17.749 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:17.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-08-30T19:31:17.795 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:19.858 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-08-30T19:31:19.858 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-08-30T19:31:19.858 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100000000 : using 0x5b8000(5.7 MiB) 2024-08-30T19:31:19.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-08-30T19:31:19.888 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:23.052 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-08-30T19:31:23.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:31:24.953 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:24.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-08-30T19:31:25.001 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-30T19:31:28.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:31:30.068 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:30.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-08-30T19:31:30.117 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:33.060 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-08-30T19:31:33.061 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-08-30T19:31:33.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:31:34.444 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:34.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-08-30T19:31:34.483 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:35.336 INFO:tasks.workunit.client.0.smithi079.stderr:Migrate to WAL device isn't supported. 2024-08-30T19:31:35.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-08-30T19:31:35.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:31:37.713 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:37.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-08-30T19:31:37.759 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:40.661 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-08-30T19:31:40.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:31:42.552 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-08-30T19:31:42.596 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:31:44.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:31:46.612 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:31:46.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:31:46.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:31:46.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:31:46.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:31:46.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:31:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:31:46.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:31:46.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:31:46.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:31:46.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:31:46.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:31:46.640 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:31:46.640 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:31:46.640 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:31:46.640 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:31:46.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:31:46.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:31:46.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:31:46.642 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:31:46.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:31:46.678 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:46.679+0000 7f60e60127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:46.681 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:46.683+0000 7f60e60127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:46.682 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:46.683+0000 7f60e60127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:47.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:31:47.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:31:47.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:31:47.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:31:47.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:31:47.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:47.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:31:47.067 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:31:47.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:47.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:31:47.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:48.191 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:48.191+0000 7f60e60127c0 -1 Falling back to public interface 2024-08-30T19:31:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:31:48.521 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:31:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:48.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:31:48.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:49.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:49.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:49.981 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:31:49.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:31:49.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:49.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:31:50.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:51.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:51.381 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:31:51.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:51.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:31:51.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:51.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:31:51.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:52.296 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:52.295+0000 7f60e60127c0 -1 osd.0 26 log_to_monitors true 2024-08-30T19:31:52.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:52.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:52.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:31:52.804 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:31:52.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:52.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:31:53.266 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 37 up_thru 37 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/3447957737,v1:127.0.0.1:6803/3447957737] [v2:127.0.0.1:6804/3447957737,v1:127.0.0.1:6805/3447957737] exists,up bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:31:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:31:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:31:53.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:31:53.267 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=119921 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-30T19:31:53.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:31:53.271 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:31:53.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:31:53.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-30T19:31:53.274 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:31:53.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:31:53.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:31:53.305 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:53.307+0000 7ff18e70c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:53.307 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:53.307+0000 7ff18e70c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:53.309 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:53.311+0000 7ff18e70c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:53.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:53.693 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:31:53.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:31:53.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:53.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:31:54.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:54.627 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:54.627+0000 7ff18e70c7c0 -1 Falling back to public interface 2024-08-30T19:31:55.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:55.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:55.126 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:31:55.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:31:55.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:55.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:31:55.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:56.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:56.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:56.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:31:56.569 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:31:56.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:56.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:31:57.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:58.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:58.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:58.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:31:58.033 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:31:58.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:58.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:31:58.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:31:58.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:58.735+0000 7ff18e70c7c0 -1 osd.1 29 log_to_monitors true 2024-08-30T19:31:59.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:31:59.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:31:59.443 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:31:59.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:31:59.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:31:59.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:31:59.942 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 40 up_thru 27 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/3089340933,v1:127.0.0.1:6811/3089340933] [v2:127.0.0.1:6812/3089340933,v1:127.0.0.1:6813/3089340933] exists,up 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:31:59.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:31:59.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:31:59.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:31:59.943 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=120894 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:31:59.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:31:59.947 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:31:59.948 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:31:59.948 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:31:59.948 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:59.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:31:59.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:31:59.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:31:59.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-30T19:31:59.950 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:31:59.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:31:59.978 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:59.979+0000 7f378b0597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:59.980 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:59.983+0000 7f378b0597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:31:59.981 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:31:59.983+0000 7f378b0597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:32:00.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:32:00.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:32:00.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:32:00.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:32:00.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:32:00.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:00.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:32:00.396 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:32:00.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:00.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:32:00.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:01.027 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:01.027+0000 7f378b0597c0 -1 Falling back to public interface 2024-08-30T19:32:01.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:01.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:01.832 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:32:01.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:32:01.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:01.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:32:02.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:03.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:03.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:03.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:32:03.237 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:32:03.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:03.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:32:03.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:04.285 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:04.287+0000 7f378b0597c0 -1 osd.2 32 log_to_monitors true 2024-08-30T19:32:04.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:04.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:04.705 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:32:04.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:32:04.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:04.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:32:05.279 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 44 up_thru 30 down_at 33 last_clean_interval [15,32) [v2:127.0.0.1:6818/3177684481,v1:127.0.0.1:6819/3177684481] [v2:127.0.0.1:6820/3177684481,v1:127.0.0.1:6821/3177684481] exists,up ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:32:05.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:32:05.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:32:05.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:32:05.280 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-30T19:32:05.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=121867 2024-08-30T19:32:05.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:32:05.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:32:05.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-30T19:32:05.289 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:32:05.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:32:05.321 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:05.323+0000 7fa4d31c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:32:05.323 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:05.323+0000 7fa4d31c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:32:05.325 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:05.327+0000 7fa4d31c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:32:06.031 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:32:06.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:06.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:32:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:06.895 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:06.895+0000 7fa4d31c27c0 -1 Falling back to public interface 2024-08-30T19:32:07.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:07.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:07.540 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:32:07.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:32:07.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:07.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:32:08.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:32:09.011 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:32:09.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:09.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:32:09.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:10.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:10.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:10.475 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:32:10.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:32:10.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:10.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:32:10.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:32:11.149 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:32:11.151+0000 7fa4d31c27c0 -1 osd.3 34 log_to_monitors true 2024-08-30T19:32:11.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:32:11.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:32:11.900 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:32:11.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:32:11.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:32:11.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:32:12.587 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/749343321,v1:127.0.0.1:6827/749343321] [v2:127.0.0.1:6828/749343321,v1:127.0.0.1:6829/749343321] exists,up fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:32:12.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:32:12.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:32:12.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:32:12.589 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=122810 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:32:12.590 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:32:12.591 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:32:12.591 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:32:12.591 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:32:12.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:32:12.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:32:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:32:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:32:12.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:32:12.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:32:12.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:32:12.702 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:32:13.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:32:13.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:32:13.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789959 2024-08-30T19:32:13.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789959 2024-08-30T19:32:13.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789959' 2024-08-30T19:32:13.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:32:13.406 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:32:13.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-08-30T19:32:13.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-08-30T19:32:13.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789959 1-171798691845' 2024-08-30T19:32:13.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:32:13.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:32:13.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561028 2024-08-30T19:32:13.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561028 2024-08-30T19:32:13.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789959 1-171798691845 2-188978561028' 2024-08-30T19:32:13.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:32:13.698 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-30T19:32:13.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397507 2024-08-30T19:32:13.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397507 2024-08-30T19:32:13.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789959 1-171798691845 2-188978561028 3-210453397507' 2024-08-30T19:32:13.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:32:13.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-158913789959 2024-08-30T19:32:13.889 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:32:13.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:32:13.891 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-158913789959 2024-08-30T19:32:13.891 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:32:13.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789959 2024-08-30T19:32:13.892 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 158913789959 2024-08-30T19:32:13.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 158913789959' 2024-08-30T19:32:13.892 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:32:14.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789957 -lt 158913789959 2024-08-30T19:32:14.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:32:15.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:32:15.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:32:15.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789959 -lt 158913789959 2024-08-30T19:32:15.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:32:15.674 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-08-30T19:32:15.674 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:32:15.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:32:15.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-08-30T19:32:15.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:32:15.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-08-30T19:32:15.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-08-30T19:32:15.678 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 171798691845 2024-08-30T19:32:15.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:32:16.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691845 2024-08-30T19:32:16.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:32:16.079 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-188978561028 2024-08-30T19:32:16.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:32:16.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:32:16.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-188978561028 2024-08-30T19:32:16.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:32:16.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561028 2024-08-30T19:32:16.083 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 188978561028 2024-08-30T19:32:16.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 188978561028' 2024-08-30T19:32:16.084 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:32:16.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561029 -lt 188978561028 2024-08-30T19:32:16.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:32:16.486 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397507 2024-08-30T19:32:16.486 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:32:16.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-30T19:32:16.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397507 2024-08-30T19:32:16.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:32:16.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397507 2024-08-30T19:32:16.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-08-30T19:32:16.489 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.3 seq 210453397507 2024-08-30T19:32:16.490 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-30T19:32:16.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397507 -lt 210453397507 2024-08-30T19:32:16.974 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:32:16.975 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:32:16.975 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:32:17.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-30T19:32:17.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:32:17.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:32:17.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-30T19:32:17.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:32:17.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:32:17.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:32:18.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-30T19:32:18.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:32:18.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:32:18.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-30T19:32:18.509 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:32:18.509 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-30T19:32:18.509 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_124151 2024-08-30T19:32:18.509 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:32:18.509 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:32:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 70 54 0.210924 0.210938 0.225664 0.246741 2024-08-30T19:32:20.509 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 134 118 0.230449 0.25 0.126515 0.250725 2024-08-30T19:32:21.509 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 188 172 0.223938 0.210938 0.361293 0.260419 2024-08-30T19:32:22.509 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 243 227 0.221657 0.214844 0.227611 0.268768 2024-08-30T19:32:23.509 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 306 290 0.226539 0.246094 0.156868 0.265528 2024-08-30T19:32:24.509 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 370 354 0.230445 0.25 0.18448 0.26409 2024-08-30T19:32:25.509 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 430 414 0.231003 0.234375 0.183677 0.263053 2024-08-30T19:32:26.509 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 493 477 0.232886 0.246094 0.159685 0.263546 2024-08-30T19:32:27.510 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 547 531 0.230443 0.210938 0.224379 0.26709 2024-08-30T19:32:28.510 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 612 596 0.232787 0.253906 0.180641 0.264809 2024-08-30T19:32:29.510 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 676 660 0.234348 0.25 0.188698 0.264378 2024-08-30T19:32:30.510 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 737 721 0.234673 0.238281 0.154052 0.263303 2024-08-30T19:32:31.510 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 801 785 0.235849 0.25 0.16169 0.261295 2024-08-30T19:32:32.510 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 855 839 0.234067 0.210938 0.203075 0.264986 2024-08-30T19:32:33.510 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 918 902 0.234867 0.246094 0.230055 0.26337 2024-08-30T19:32:34.510 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 991 975 0.238008 0.285156 0.20097 0.260482 2024-08-30T19:32:35.511 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1077 1061 0.243766 0.335938 0.124615 0.255151 2024-08-30T19:32:36.511 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1146 1130 0.245195 0.269531 0.210345 0.252734 2024-08-30T19:32:37.511 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1209 1193 0.245241 0.246094 0.178466 0.251058 2024-08-30T19:32:38.511 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:32:38.515132+0000 min lat: 0.100545 max lat: 0.470018 avg lat: 0.251058 2024-08-30T19:32:38.511 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:32:38.511 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1286 1270 0.248016 0.300781 0.214604 0.251058 2024-08-30T19:32:39.511 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1367 1351 0.25127 0.316406 0.180936 0.247848 2024-08-30T19:32:40.511 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1447 1431 0.254051 0.3125 0.211413 0.245342 2024-08-30T19:32:41.511 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1528 1512 0.256761 0.316406 0.197408 0.241697 2024-08-30T19:32:42.512 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1606 1590 0.258756 0.304688 0.208442 0.240411 2024-08-30T19:32:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1674 1658 0.25903 0.265625 0.310319 0.239723 2024-08-30T19:32:44.512 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1765 1749 0.262737 0.355469 0.155892 0.237563 2024-08-30T19:32:45.512 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1835 1819 0.263132 0.273438 0.154189 0.23651 2024-08-30T19:32:46.512 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1912 1896 0.264474 0.300781 0.160627 0.235431 2024-08-30T19:32:47.512 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1990 1974 0.26586 0.304688 0.17932 0.234694 2024-08-30T19:32:48.513 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 2063 2047 0.266502 0.285156 0.22982 0.233545 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1539 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 2063 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.267249 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0396598 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.355469 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.210938 2024-08-30T19:32:49.513 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 68 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 10.1529 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 91 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 54 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.233384 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0624815 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.470018 2024-08-30T19:32:49.514 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0973601 2024-08-30T19:32:49.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119921 2024-08-30T19:32:49.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-08-30T19:32:50.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119921 2024-08-30T19:32:50.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (119921) - No such process 2024-08-30T19:32:50.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-08-30T19:32:51.135 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:32:51.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120894 2024-08-30T19:32:51.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-08-30T19:32:52.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120894 2024-08-30T19:32:52.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (120894) - No such process 2024-08-30T19:32:52.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-08-30T19:32:52.700 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:32:52.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121867 2024-08-30T19:32:52.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-08-30T19:32:53.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121867 2024-08-30T19:32:53.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (121867) - No such process 2024-08-30T19:32:53.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-08-30T19:32:54.172 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:32:54.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122810 2024-08-30T19:32:54.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-08-30T19:32:55.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122810 2024-08-30T19:32:55.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (122810) - No such process 2024-08-30T19:32:55.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-08-30T19:32:55.723 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-08-30T19:32:55.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:32:57.604 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:32:57.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-08-30T19:32:57.907 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records in 2024-08-30T19:32:57.907 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records out 2024-08-30T19:32:57.907 INFO:tasks.workunit.client.0.smithi079.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.279993 s, 1.9 GB/s 2024-08-30T19:32:57.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-08-30T19:32:57.931 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:03.572 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-08-30T19:33:03.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:33:05.945 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:05.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:33:07.824 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:07.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-08-30T19:33:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:33:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:33:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.58725 s, 1.8 GB/s 2024-08-30T19:33:08.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-08-30T19:33:08.453 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:17.077 INFO:tasks.workunit.client.0.smithi079.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-08-30T19:33:17.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-08-30T19:33:17.108 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:20.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:33:22.632 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:22.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:33:24.025 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:24.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-08-30T19:33:24.057 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-30T19:33:24.057 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:25.416 INFO:tasks.workunit.client.0.smithi079.stdout:DB device added 2024-08-30T19:33:25.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-08-30T19:33:25.448 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-30T19:33:25.448 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:27.576 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added 2024-08-30T19:33:27.583 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-08-30T19:33:27.607 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:30.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:33:33.145 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:33.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:33:35.036 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:35.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-08-30T19:33:35.078 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-30T19:33:35.078 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:37.216 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added 2024-08-30T19:33:37.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-08-30T19:33:37.247 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:33:40.661 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-08-30T19:33:40.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:33:42.553 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:33:42.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:33:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:33:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:33:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:33:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:33:42.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:33:42.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:42.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:42.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:33:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:33:42.575 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:33:42.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:33:42.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:33:42.576 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-30T19:33:42.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:33:42.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:33:42.578 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:33:42.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:33:42.607 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:42.605+0000 7f35dcf5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:42.610 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:42.609+0000 7f35dcf5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:42.611 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:42.609+0000 7f35dcf5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:42.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:33:42.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:33:42.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:42.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:43.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:44.347 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:44.345+0000 7f35dcf5c7c0 -1 Falling back to public interface 2024-08-30T19:33:44.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:44.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:44.415 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:33:44.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:33:44.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:44.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:44.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:45.817 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:33:45.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:45.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:45.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:33:45.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:45.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:46.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:47.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:47.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:48.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:33:48.702 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:33:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:48.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:49.024 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:49.025+0000 7f35dcf5c7c0 -1 osd.0 51 log_to_monitors true 2024-08-30T19:33:49.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:50.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:50.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:50.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:33:50.165 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:33:50.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:50.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:33:50.585 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 62 up_thru 62 down_at 52 last_clean_interval [37,51) [v2:127.0.0.1:6802/2454588068,v1:127.0.0.1:6803/2454588068] [v2:127.0.0.1:6804/2454588068,v1:127.0.0.1:6805/2454588068] exists,up bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:33:50.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:33:50.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:33:50.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:33:50.586 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:33:50.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=130302 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-30T19:33:50.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-30T19:33:50.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:33:50.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:33:50.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:33:50.590 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:33:50.590 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:33:50.590 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:33:50.590 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:33:50.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:33:50.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:33:50.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-30T19:33:50.592 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:33:50.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:33:50.621 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:50.621+0000 7f03e52407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:50.646 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:50.645+0000 7f03e52407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:50.647 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:50.645+0000 7f03e52407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:51.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:33:51.040 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:33:51.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:51.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:51.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:52.363 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:52.361+0000 7f03e52407c0 -1 Falling back to public interface 2024-08-30T19:33:52.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:52.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:52.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:33:52.457 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:33:52.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:52.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:52.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:53.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:53.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:53.857 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:33:53.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:33:53.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:53.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:54.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:55.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:55.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:55.260 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:33:55.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:33:55.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:55.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:55.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:56.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:57.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:33:57.466 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:57.464+0000 7f03e52407c0 -1 osd.1 54 log_to_monitors true 2024-08-30T19:33:58.101 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:33:58.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:33:58.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:58.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:33:58.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:58.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:33:58.680 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [40,54) [v2:127.0.0.1:6810/1084131450,v1:127.0.0.1:6811/1084131450] [v2:127.0.0.1:6812/1084131450,v1:127.0.0.1:6813/1084131450] exists,up 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:33:58.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:33:58.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:33:58.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:33:58.681 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-30T19:33:58.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=131305 2024-08-30T19:33:58.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-30T19:33:58.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:33:58.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:33:58.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:33:58.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:33:58.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:33:58.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:33:58.702 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:33:58.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-30T19:33:58.704 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:33:58.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:33:58.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:33:58.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:58.732+0000 7fed331977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:58.780 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:58.776+0000 7fed331977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:58.782 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:33:58.780+0000 7fed331977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:33:59.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:33:59.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:33:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:33:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:33:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:33:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:33:59.179 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:33:59.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:33:59.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:33:59.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:33:59.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:00.247 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:00.244+0000 7fed331977c0 -1 Falling back to public interface 2024-08-30T19:34:00.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:00.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:00.621 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:34:00.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:34:00.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:00.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:34:01.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:02.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:02.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:02.031 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:34:02.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:34:02.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:02.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:34:02.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:03.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:03.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:03.445 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:34:03.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:34:03.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:03.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:34:03.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:04.873 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:34:04.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:04.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:04.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:34:04.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:04.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:34:05.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:05.906 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:05.904+0000 7fed331977c0 -1 osd.2 57 log_to_monitors true 2024-08-30T19:34:06.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:06.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:06.308 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:34:06.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:34:06.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:06.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:34:06.855 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 68 up_thru 55 down_at 58 last_clean_interval [44,57) [v2:127.0.0.1:6818/3563700700,v1:127.0.0.1:6819/3563700700] [v2:127.0.0.1:6820/3563700700,v1:127.0.0.1:6821/3563700700] exists,up ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:34:06.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:34:06.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:34:06.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:34:06.856 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=132308 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-30T19:34:06.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:34:06.860 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:34:06.861 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:34:06.861 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:34:06.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-30T19:34:06.863 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:34:06.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-30T19:34:06.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:34:06.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-30T19:34:06.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-30T19:34:06.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:34:06.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:34:06.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:34:06.886 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:06.884+0000 7fc2f69247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:34:06.888 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:06.884+0000 7fc2f69247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:34:06.890 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:06.888+0000 7fc2f69247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:07.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:34:07.524 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:34:07.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:07.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:34:08.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:08.179 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:08.176+0000 7fc2f69247c0 -1 Falling back to public interface 2024-08-30T19:34:09.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:09.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:09.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:34:09.095 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:34:09.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:09.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:34:09.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:10.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:10.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:10.528 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:34:10.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:34:10.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:10.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:34:10.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:11.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:11.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:11.943 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:34:11.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:34:11.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:11.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:34:12.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:34:12.810 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:34:12.808+0000 7fc2f69247c0 -1 osd.3 59 log_to_monitors true 2024-08-30T19:34:13.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:34:13.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:34:13.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:34:13.387 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:34:13.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:34:13.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:34:14.069 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 73 up_thru 58 down_at 60 last_clean_interval [49,59) [v2:127.0.0.1:6826/672526349,v1:127.0.0.1:6827/672526349] [v2:127.0.0.1:6828/672526349,v1:127.0.0.1:6829/672526349] exists,up fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:34:14.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:34:14.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:34:14.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:34:14.070 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-30T19:34:14.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=133311 2024-08-30T19:34:14.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-30T19:34:14.108 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:34:14.108 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-30T19:34:14.108 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_134269 2024-08-30T19:34:14.108 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:34:14.108 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:34:15.108 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 22 6 0.023434 0.0234375 0.270135 0.243862 2024-08-30T19:34:16.108 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 49 33 0.0644447 0.105469 0.351361 0.882786 2024-08-30T19:34:17.108 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 101 85 0.110665 0.203125 0.217281 0.537361 2024-08-30T19:34:18.108 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 148 132 0.128892 0.183594 0.34997 0.463907 2024-08-30T19:34:19.109 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 203 187 0.146078 0.214844 0.228591 0.414882 2024-08-30T19:34:20.109 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 251 235 0.152978 0.1875 0.275142 0.398634 2024-08-30T19:34:21.109 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 297 281 0.156791 0.179688 0.252671 0.385129 2024-08-30T19:34:22.109 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 347 331 0.161603 0.195312 0.308638 0.377849 2024-08-30T19:34:23.109 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 392 376 0.163177 0.175781 0.380881 0.373931 2024-08-30T19:34:24.109 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 445 429 0.16756 0.207031 0.289318 0.366653 2024-08-30T19:34:25.109 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 492 476 0.169015 0.183594 0.213172 0.36329 2024-08-30T19:34:26.109 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 537 521 0.169577 0.175781 0.273547 0.36331 2024-08-30T19:34:27.109 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 585 569 0.170954 0.1875 0.343507 0.361711 2024-08-30T19:34:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 636 620 0.172972 0.199219 0.34113 0.358604 2024-08-30T19:34:29.110 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 683 667 0.173678 0.183594 0.326982 0.356602 2024-08-30T19:34:30.110 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 734 718 0.175273 0.199219 0.351945 0.352357 2024-08-30T19:34:31.110 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 775 759 0.174383 0.160156 0.375216 0.356196 2024-08-30T19:34:32.110 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 820 804 0.174459 0.175781 0.302009 0.354187 2024-08-30T19:34:33.110 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 877 861 0.176995 0.222656 0.309656 0.350926 2024-08-30T19:34:34.110 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:34:34.114468+0000 min lat: 0.142491 max lat: 1.83835 avg lat: 0.350075 2024-08-30T19:34:34.110 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:34:34.111 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 920 904 0.176542 0.167969 0.36862 0.350075 2024-08-30T19:34:35.110 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 975 959 0.178365 0.214844 0.211509 0.34774 2024-08-30T19:34:36.110 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1018 1002 0.177891 0.167969 0.338786 0.347551 2024-08-30T19:34:37.111 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1076 1060 0.180006 0.226562 0.308868 0.344845 2024-08-30T19:34:38.111 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1120 1104 0.179666 0.171875 0.212534 0.344588 2024-08-30T19:34:39.111 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1171 1155 0.180447 0.199219 0.367415 0.342712 2024-08-30T19:34:40.111 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1226 1210 0.181769 0.214844 0.334854 0.342364 2024-08-30T19:34:41.111 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1267 1251 0.180968 0.160156 0.476577 0.342877 2024-08-30T19:34:42.111 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1311 1295 0.180642 0.171875 0.225808 0.343595 2024-08-30T19:34:43.111 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1363 1347 0.181416 0.203125 0.284575 0.342248 2024-08-30T19:34:44.112 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 1413 1397 0.181879 0.195312 0.299989 0.341777 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1558 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 1413 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.183034 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0382277 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.226562 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.0234375 2024-08-30T19:34:45.112 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 46 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 9.78628 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 58 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 6 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.340783 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.130916 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 1.83835 2024-08-30T19:34:45.113 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.142491 2024-08-30T19:34:45.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130302 2024-08-30T19:34:45.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-08-30T19:34:46.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130302 2024-08-30T19:34:46.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (130302) - No such process 2024-08-30T19:34:46.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-08-30T19:34:46.868 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:34:46.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131305 2024-08-30T19:34:46.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-08-30T19:34:47.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131305 2024-08-30T19:34:47.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (131305) - No such process 2024-08-30T19:34:47.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-08-30T19:34:48.343 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-08-30T19:34:48.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132308 2024-08-30T19:34:48.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-08-30T19:34:49.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132308 2024-08-30T19:34:49.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (132308) - No such process 2024-08-30T19:34:49.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-08-30T19:34:49.811 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-08-30T19:34:49.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133311 2024-08-30T19:34:49.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-08-30T19:34:50.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133311 2024-08-30T19:34:50.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (133311) - No such process 2024-08-30T19:34:50.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-08-30T19:34:51.536 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-08-30T19:34:51.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:34:53.916 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:34:53.938 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-08-30T19:34:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:34:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:34:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.560561 s, 1.9 GB/s 2024-08-30T19:34:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-08-30T19:34:54.525 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:05.464 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-08-30T19:35:05.465 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-08-30T19:35:05.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:35:07.844 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:07.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-08-30T19:35:08.164 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records in 2024-08-30T19:35:08.164 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records out 2024-08-30T19:35:08.165 INFO:tasks.workunit.client.0.smithi079.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.299003 s, 1.8 GB/s 2024-08-30T19:35:08.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-08-30T19:35:08.189 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:15.108 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-08-30T19:35:15.108 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-08-30T19:35:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-08-30T19:35:15.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-30T19:35:17.548 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:17.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:35:19.940 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:19.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-08-30T19:35:20.527 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:35:20.527 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:35:20.527 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.561009 s, 1.9 GB/s 2024-08-30T19:35:20.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-08-30T19:35:20.551 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:31.464 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-30T19:35:31.464 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-08-30T19:35:31.465 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-08-30T19:35:31.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-08-30T19:35:31.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:35:33.468 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:33.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-08-30T19:35:33.512 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:35.652 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-30T19:35:35.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-08-30T19:35:35.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-30T19:35:37.156 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:37.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-08-30T19:35:39.268 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:39.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-08-30T19:35:39.855 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:35:39.855 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:35:39.855 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.562243 s, 1.9 GB/s 2024-08-30T19:35:39.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-08-30T19:35:39.877 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:40.728 INFO:tasks.workunit.client.0.smithi079.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-08-30T19:35:40.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-08-30T19:35:40.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:35:43.108 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:43.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-08-30T19:35:51.345 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:35:51.345 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:35:51.346 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.07501 s, 133 MB/s 2024-08-30T19:35:51.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-08-30T19:35:51.371 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:35:55.032 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-08-30T19:35:55.032 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-08-30T19:35:55.032 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-08-30T19:35:55.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-30T19:35:56.952 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:35:56.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-08-30T19:35:57.548 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-08-30T19:35:57.548 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-08-30T19:35:57.548 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.580615 s, 1.8 GB/s 2024-08-30T19:35:57.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-08-30T19:35:57.573 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:36:07.456 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-08-30T19:36:07.456 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-08-30T19:36:07.457 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-08-30T19:36:07.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-30T19:36:09.336 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:36:09.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:36:09.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:36:09.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:36:09.357 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:36:09.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:36:09.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:36:09.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:36:09.359 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:36:09.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:36:09.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:36:09.360 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-30T19:36:09.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:36:09.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:36:09.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:36:09.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:36:09.395 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:09.394+0000 7f1ff09df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:09.398 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:09.398+0000 7f1ff09df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:09.399 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:09.398+0000 7f1ff09df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:36:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:09.838 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:36:09.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:36:09.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:09.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:10.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:10.899 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:10.898+0000 7f1ff09df7c0 -1 Falling back to public interface 2024-08-30T19:36:11.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:11.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:11.255 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:36:11.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:36:11.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:11.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:11.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:12.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:12.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:12.667 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:36:12.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:36:12.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:12.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:13.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:14.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:14.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:14.078 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:36:14.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:36:14.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:14.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:14.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:15.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:15.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:15.501 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:36:15.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:36:15.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:15.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:15.859 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:15.858+0000 7f1ff09df7c0 -1 osd.0 75 log_to_monitors true 2024-08-30T19:36:15.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:16.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:16.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:16.928 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:36:16.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:36:16.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:16.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:36:17.392 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [62,75) [v2:127.0.0.1:6802/3020561411,v1:127.0.0.1:6803/3020561411] [v2:127.0.0.1:6804/3020561411,v1:127.0.0.1:6805/3020561411] exists,up bbc0aa84-6359-4dfc-a989-1b1ef0fe5f66 2024-08-30T19:36:17.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:36:17.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:36:17.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:36:17.393 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:36:17.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=140447 2024-08-30T19:36:17.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:36:17.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:36:17.396 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:36:17.396 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:36:17.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:36:17.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:36:17.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:36:17.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:36:17.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-30T19:36:17.398 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:36:17.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:36:17.427 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:17.426+0000 7f925ba1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:17.463 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:17.462+0000 7f925ba1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:17.465 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:17.466+0000 7f925ba1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:17.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-30T19:36:17.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:36:17.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:17.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:36:18.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:18.714 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:18.714+0000 7f925ba1f7c0 -1 Falling back to public interface 2024-08-30T19:36:19.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:19.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:19.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:36:19.200 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:36:19.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:19.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:36:19.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:20.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:20.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:20.655 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:36:20.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:36:20.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:20.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:36:21.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:22.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:22.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:22.055 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:36:22.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:36:22.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:22.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-30T19:36:22.072 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:22.074+0000 7f925ba1f7c0 -1 osd.1 78 log_to_monitors true 2024-08-30T19:36:22.512 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 89 up_thru 76 down_at 79 last_clean_interval [65,78) [v2:127.0.0.1:6810/2475489880,v1:127.0.0.1:6811/2475489880] [v2:127.0.0.1:6812/2475489880,v1:127.0.0.1:6813/2475489880] exists,up 9bbb26fc-0d11-4ab6-8181-d22788708029 2024-08-30T19:36:22.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:36:22.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:36:22.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:36:22.513 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-30T19:36:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=141450 2024-08-30T19:36:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-30T19:36:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:36:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:36:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:36:22.517 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:36:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:36:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:36:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:36:22.518 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:36:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-30T19:36:22.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-30T19:36:22.520 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-08-30T19:36:22.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:36:22.521 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-30T19:36:22.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-30T19:36:22.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:36:22.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:36:22.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:36:22.548 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:22.546+0000 7f42e4c487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:22.622 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:22.622+0000 7f42e4c487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:22.623 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:22.622+0000 7f42e4c487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:23.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:23.010 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:36:23.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:36:23.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:23.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:36:23.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:24.154 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:24.154+0000 7f42e4c487c0 -1 Falling back to public interface 2024-08-30T19:36:24.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:24.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:24.535 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:36:24.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:36:24.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:24.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:36:24.941 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:25.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:25.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:25.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:36:25.959 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:36:25.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:25.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:36:26.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:27.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:27.353 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:36:27.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:27.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:36:27.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:36:27.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:27.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:28.292 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:28.294+0000 7f42e4c487c0 -1 osd.2 81 log_to_monitors true 2024-08-30T19:36:28.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:28.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:28.783 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:36:28.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:36:28.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:28.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-30T19:36:29.547 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 93 up_thru 79 down_at 82 last_clean_interval [68,81) [v2:127.0.0.1:6818/2269665908,v1:127.0.0.1:6819/2269665908] [v2:127.0.0.1:6820/2269665908,v1:127.0.0.1:6821/2269665908] exists,up ae53fa2b-f9b9-48d3-a19a-4f3cdea04088 2024-08-30T19:36:29.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:36:29.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:36:29.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:36:29.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=142410 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:36:29.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:36:29.556 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:36:29.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:36:29.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:36:29.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-30T19:36:29.558 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:36:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:36:29.589 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:29.589+0000 7fd78b47b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:29.646 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:29.645+0000 7fd78b47b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:29.648 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:29.649+0000 7fd78b47b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:30.127 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:36:30.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:36:30.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:30.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:36:30.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:30.698 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:30.697+0000 7fd78b47b7c0 -1 Falling back to public interface 2024-08-30T19:36:31.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:31.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:31.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:36:31.616 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:36:31.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:31.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:36:32.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:33.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:33.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:33.040 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:36:33.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:36:33.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:33.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:36:33.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:34.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:34.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:34.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:36:34.453 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:36:34.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:34.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:36:34.808 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:36:34.805+0000 7fd78b47b7c0 -1 osd.3 83 log_to_monitors true 2024-08-30T19:36:34.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:36:35.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:36:35.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:36:35.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:36:35.896 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:36:35.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:36:35.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-30T19:36:36.773 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 98 up_thru 98 down_at 84 last_clean_interval [73,83) [v2:127.0.0.1:6826/1063352214,v1:127.0.0.1:6827/1063352214] [v2:127.0.0.1:6828/1063352214,v1:127.0.0.1:6829/1063352214] exists,up fd702c6c-ce07-4321-8699-4c23c2bd6194 2024-08-30T19:36:36.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:36:36.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:36:36.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:36:36.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-30T19:36:36.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=143366 2024-08-30T19:36:36.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-30T19:36:36.814 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:36:36.814 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-30T19:36:36.814 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_144324 2024-08-30T19:36:36.814 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:36:36.814 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:36:37.814 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 75 59 0.230435 0.230469 0.186381 0.236535 2024-08-30T19:36:38.814 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 157 141 0.275349 0.320312 0.169479 0.207429 2024-08-30T19:36:39.814 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 247 231 0.300733 0.351562 0.215343 0.204089 2024-08-30T19:36:40.814 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 331 315 0.307573 0.328125 0.125153 0.19548 2024-08-30T19:36:41.814 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 430 414 0.323393 0.386719 0.191795 0.189133 2024-08-30T19:36:42.815 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 527 511 0.332639 0.378906 0.181722 0.184807 2024-08-30T19:36:43.815 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 621 605 0.337567 0.367188 0.205058 0.183799 2024-08-30T19:36:44.815 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 723 707 0.34517 0.398438 0.161323 0.179999 2024-08-30T19:36:45.815 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 824 808 0.350648 0.394531 0.122873 0.176653 2024-08-30T19:36:46.815 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 936 920 0.359327 0.4375 0.157606 0.172746 2024-08-30T19:36:47.815 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 1027 1011 0.358973 0.355469 0.167609 0.173245 2024-08-30T19:36:48.815 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 1131 1115 0.362908 0.40625 0.164733 0.17128 2024-08-30T19:36:49.816 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 1223 1207 0.362633 0.359375 0.190702 0.170464 2024-08-30T19:36:50.816 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 1299 1283 0.357934 0.296875 0.133889 0.173911 2024-08-30T19:36:51.816 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 1383 1367 0.355943 0.328125 0.175947 0.174229 2024-08-30T19:36:52.816 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 1493 1477 0.360547 0.429688 0.100843 0.172613 2024-08-30T19:36:53.816 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1589 1573 0.361395 0.375 0.207906 0.172134 2024-08-30T19:36:54.816 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1701 1685 0.36562 0.4375 0.143398 0.170104 2024-08-30T19:36:55.816 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1771 1755 0.360767 0.273438 0.367596 0.172121 2024-08-30T19:36:56.817 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:36:56.820878+0000 min lat: 0.0781401 max lat: 0.689662 avg lat: 0.171614 2024-08-30T19:36:56.817 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:36:56.817 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1874 1858 0.362844 0.402344 0.123567 0.171614 2024-08-30T19:36:57.817 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1981 1965 0.365466 0.417969 0.125606 0.170429 2024-08-30T19:36:58.817 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 2085 2069 0.367318 0.40625 0.152964 0.169665 2024-08-30T19:36:59.817 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 2169 2153 0.365613 0.328125 0.167061 0.169979 2024-08-30T19:37:00.817 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 2271 2255 0.366978 0.398438 0.144366 0.169196 2024-08-30T19:37:01.817 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 2338 2322 0.362766 0.261719 0.171606 0.171484 2024-08-30T19:37:02.817 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 2436 2420 0.363536 0.382812 0.13652 0.171228 2024-08-30T19:37:03.817 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 2531 2515 0.363814 0.371094 0.201041 0.171304 2024-08-30T19:37:04.818 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 2633 2617 0.365048 0.398438 0.157107 0.170562 2024-08-30T19:37:05.818 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 2726 2710 0.364985 0.363281 0.232796 0.170459 2024-08-30T19:37:06.818 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 2822 2806 0.365317 0.375 0.155757 0.170463 2024-08-30T19:37:07.818 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1032 2024-08-30T19:37:07.818 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 2822 2024-08-30T19:37:07.818 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-08-30T19:37:07.818 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.366188 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0511656 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.4375 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.230469 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 93 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 13.0984 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 112 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 59 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.17043 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0524442 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.689662 2024-08-30T19:37:07.819 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0781401 2024-08-30T19:37:07.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-08-30T19:37:07.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:37:07.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:37:07.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:37:07.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:37:07.882 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:37:07.883 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:37:07.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:37:07.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:37:07.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:37:08.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:37:08.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:37:08.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:37:08.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:37:08.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:37:08.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:37:08.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:37:08.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187469 2024-08-30T19:37:08.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187469 2024-08-30T19:37:08.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-369367187469' 2024-08-30T19:37:08.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:37:08.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-30T19:37:08.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089356 2024-08-30T19:37:08.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089356 2024-08-30T19:37:08.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-08-30T19:37:08.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:37:08.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-30T19:37:08.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958539 2024-08-30T19:37:08.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958539 2024-08-30T19:37:08.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539' 2024-08-30T19:37:08.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:37:08.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-30T19:37:09.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795017 2024-08-30T19:37:09.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795017 2024-08-30T19:37:09.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539 3-420906795017' 2024-08-30T19:37:09.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:37:09.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-369367187469 2024-08-30T19:37:09.043 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:37:09.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:37:09.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-369367187469 2024-08-30T19:37:09.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:37:09.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187469 2024-08-30T19:37:09.045 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 369367187469 2024-08-30T19:37:09.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-08-30T19:37:09.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:37:09.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187469 -lt 369367187469 2024-08-30T19:37:09.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:37:09.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089356 2024-08-30T19:37:09.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:37:09.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-08-30T19:37:09.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089356 2024-08-30T19:37:09.438 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:37:09.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089356 2024-08-30T19:37:09.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-08-30T19:37:09.440 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 382252089356 2024-08-30T19:37:09.441 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-30T19:37:09.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089356 -lt 382252089356 2024-08-30T19:37:09.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:37:09.835 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-399431958539 2024-08-30T19:37:09.835 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:37:09.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-08-30T19:37:09.838 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-399431958539 2024-08-30T19:37:09.838 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:37:09.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958539 2024-08-30T19:37:09.839 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 399431958539 2024-08-30T19:37:09.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 399431958539' 2024-08-30T19:37:09.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-30T19:37:10.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958539 -lt 399431958539 2024-08-30T19:37:10.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:37:10.259 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-420906795017 2024-08-30T19:37:10.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:37:10.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-08-30T19:37:10.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-420906795017 2024-08-30T19:37:10.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:37:10.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795017 2024-08-30T19:37:10.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 420906795017' 2024-08-30T19:37:10.263 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.3 seq 420906795017 2024-08-30T19:37:10.264 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-30T19:37:10.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795015 -lt 420906795017 2024-08-30T19:37:10.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:37:11.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:37:11.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-30T19:37:12.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795018 -lt 420906795017 2024-08-30T19:37:12.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:37:12.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:37:12.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:37:12.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:37:12.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:37:13.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-30T19:37:13.078 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:37:13.079 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:37:13.079 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:37:13.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:37:13.591 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:37:13.591 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:37:13.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:37:13.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:37:13.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:37:13.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:37:13.902 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:37:13.903 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:37:13.904 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:37:13.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:37:13.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:37:13.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:37:13.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:37:13.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:37:13.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:37:13.908 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:37:13.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:37:13.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:37:13.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:37:14.358 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:37:14.358 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:14.358 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:14.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-30T19:37:14.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:14.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:37:14.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:37:14.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:37:14.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:37:14.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:37:14.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:37:14.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:37:14.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:37:14.366 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:37:14.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:37:14.367 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:37:14.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:37:14.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:37:14.369 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:37:14.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:37:14.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:37:14.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:37:14.370 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:37:14.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:37:14.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:37:14.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:37:14.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:37:14.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:14.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:14.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:37:14.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:37:14.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:37:14.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-30T19:37:14.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:37:14.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:14.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:14.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97333 2024-08-30T19:37:14.378 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:37:14.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-30T19:37:14.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:37:14.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-30T19:37:14.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-08-30T19:37:14.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-08-30T19:37:14.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-08-30T19:37:14.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-30T19:37:14.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:14.806 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:37:14.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:37:14.807 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:37:14.807 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:37:14.807 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:14.807 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:14.808 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:37:14.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:37:15.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:37:15.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:37:15.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:37:15.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:37:15.026 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:37:15.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:37:15.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:37:15.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:37:15.027 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:37:15.028 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:37:15.028 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:15.028 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:15.028 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:37:15.028 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:37:15.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get fsid 2024-08-30T19:37:15.144 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:37:15.144 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:37:15.144 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:37:15.144 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:37:15.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:37:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:37:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:37:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:37:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:37:15.146 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:15.146 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:15.146 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:37:15.146 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:37:15.146 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get mon_host 2024-08-30T19:37:15.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-08-30T19:37:15.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:15.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:37:15.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:37:15.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:37:15.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-30T19:37:15.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:37:16.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:37:16.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:37:16.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:37:16.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:37:16.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:37:16.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:37:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:37:16.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:37:16.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38709232-3c93-4c80-b8f2-4944b2e21816 2024-08-30T19:37:16.230 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 38709232-3c93-4c80-b8f2-4944b2e21816 2024-08-30T19:37:16.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 38709232-3c93-4c80-b8f2-4944b2e21816' 2024-08-30T19:37:16.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:37:16.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsH9JmRe/3DhAAkeTt/C2/JZHhkHKcg24UgQ== 2024-08-30T19:37:16.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsH9JmRe/3DhAAkeTt/C2/JZHhkHKcg24UgQ=="}' 2024-08-30T19:37:16.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38709232-3c93-4c80-b8f2-4944b2e21816 -i td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:37:16.564 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:37:16.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:37:16.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsH9JmRe/3DhAAkeTt/C2/JZHhkHKcg24UgQ== --osd-uuid 38709232-3c93-4c80-b8f2-4944b2e21816 2024-08-30T19:37:16.616 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:16.616+0000 7f771b8867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:16.640 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:16.644+0000 7f771b8867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:16.642 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:16.644+0000 7f771b8867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:16.642 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:16.644+0000 7f771b8867c0 -1 bdev(0x5589da4e2a80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:37:16.643 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:16.644+0000 7f771b8867c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-30T19:37:25.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-30T19:37:25.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:37:25.140 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:37:25.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:37:25.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:37:25.710 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:37:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:37:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:37:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:37:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:37:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:37:25.748 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:25.748+0000 7f231c2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:25.765 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:25.768+0000 7f231c2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:25.767 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:25.768+0000 7f231c2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:37:26.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:26.120 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:37:26.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:37:26.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:26.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:26.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:27.035 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:27.036+0000 7f231c2827c0 -1 Falling back to public interface 2024-08-30T19:37:27.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:27.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:27.554 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:37:27.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:37:27.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:27.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:27.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:28.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:28.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:28.978 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:37:28.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:37:28.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:28.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:29.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:30.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:30.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:30.388 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:37:30.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:37:30.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:30.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:30.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:31.457 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:37:31.459+0000 7f231c2827c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:37:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:37:31.817 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:37:31.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:31.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:33.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:34.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:34.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:34.047 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:37:34.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:37:34.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:34.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:35.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:37:36.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:37:36.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:37:36.567 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:37:36.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:37:36.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:37:36.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:37:37.015 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/924966211,v1:127.0.0.1:6803/924966211] [v2:127.0.0.1:6804/924966211,v1:127.0.0.1:6805/924966211] exists,up 38709232-3c93-4c80-b8f2-4944b2e21816 2024-08-30T19:37:37.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:37:37.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:37:37.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:37:37.016 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:37:37.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=146224 2024-08-30T19:37:37.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-08-30T19:37:42.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-08-30T19:37:42.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-30T19:37:42.574 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-08-30T19:37:42.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:37:43.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-08-30T19:37:43.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-08-30T19:37:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-08-30T19:37:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-08-30T19:37:43.636 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:37:43.638 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-08-30T19:37:43.638 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_147515 2024-08-30T19:37:43.671 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:37:43.671 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:37:44.672 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 17 1 3.9996 4 0.255042 0.255042 2024-08-30T19:37:45.672 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 33 17 33.9955 64 1.11078 1.19946 2024-08-30T19:37:46.672 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 49 33 43.9935 64 1.17788 1.15376 2024-08-30T19:37:47.672 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 65 49 48.9923 64 1.05165 1.08891 2024-08-30T19:37:48.673 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 81 65 51.9918 64 1.29459 1.18253 2024-08-30T19:37:49.673 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 81 65 43.3265 0 - 1.18253 2024-08-30T19:37:50.673 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 97 81 46.2784 32 1.45333 1.23453 2024-08-30T19:37:51.673 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 113 97 48.4923 64 1.29441 1.23327 2024-08-30T19:37:52.673 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 129 113 50.2142 64 1.05189 1.20735 2024-08-30T19:37:53.673 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 144 128 51.1918 60 1.36254 1.22496 2024-08-30T19:37:54.673 INFO:tasks.workunit.client.0.smithi079.stdout: 11 15 144 129 46.9015 4 1.36997 1.22609 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 11.0119 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 144 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4194304 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4194304 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 52.3068 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 28.1709 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 64 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 13 2024-08-30T19:37:55.674 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 7.04273 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 16 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 0 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 1.21689 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.188065 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 1.49951 2024-08-30T19:37:55.675 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.255042 2024-08-30T19:37:55.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-08-30T19:38:05.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-30T19:38:05.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-08-30T19:38:05.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1003487232 2024-08-30T19:38:05.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-08-30T19:38:05.913 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-30T19:38:06.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=0 2024-08-30T19:38:06.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-08-30T19:38:06.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 0 -eq 0 2024-08-30T19:38:06.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ 1 -le 5 ]] 2024-08-30T19:38:06.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-08-30T19:38:06.119 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:38:06.120 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-08-30T19:38:06.121 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_147588 2024-08-30T19:38:06.151 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:38:06.151 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:38:07.151 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 32 16 63.9905 64 0.935963 0.888813 2024-08-30T19:38:08.152 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 33 17 33.9947 4 0.85937 0.887081 2024-08-30T19:38:09.152 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 64 48 63.9907 124 0.964789 0.97741 2024-08-30T19:38:10.152 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 77 61 60.9914 52 1.00126 0.982919 2024-08-30T19:38:11.152 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 81 65 51.9923 16 0.959578 0.985316 2024-08-30T19:38:12.152 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 97 81 53.9918 64 1.36237 1.05154 2024-08-30T19:38:13.152 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 113 97 55.4202 64 1.08539 1.03596 2024-08-30T19:38:14.153 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 129 113 56.4913 64 1.36219 1.06554 2024-08-30T19:38:15.153 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 129 113 50.2145 0 - 1.06554 2024-08-30T19:38:16.153 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 129 113 45.1931 0 - 1.06554 2024-08-30T19:38:17.153 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 129 113 41.0847 0 - 1.06554 2024-08-30T19:38:18.153 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 11.5038 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 129 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4194304 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4194304 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 44.8547 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 40.2031 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 124 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 11 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 10.0508 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 31 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 0 2024-08-30T19:38:18.154 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 1.41424 2024-08-30T19:38:18.155 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.943334 2024-08-30T19:38:18.155 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 3.89751 2024-08-30T19:38:18.155 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.184468 2024-08-30T19:38:18.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-08-30T19:38:28.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-30T19:38:28.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-08-30T19:38:28.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=126877696 2024-08-30T19:38:28.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-30T19:38:28.402 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-08-30T19:38:28.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=535953408 2024-08-30T19:38:28.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-08-30T19:38:28.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 535953408 -eq 0 2024-08-30T19:38:28.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-08-30T19:38:28.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 535953408 -gt 0 2024-08-30T19:38:28.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146224 2024-08-30T19:38:28.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-08-30T19:38:29.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146224 2024-08-30T19:38:29.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (146224) - No such process 2024-08-30T19:38:29.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-08-30T19:38:29.991 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:38:30.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-08-30T19:38:30.034 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:38:33.657 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-08-30T19:38:33.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-08-30T19:38:33.716 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-08-30T19:38:35.278 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0xffffe000 : using 0x27a38000(634 MiB) 2024-08-30T19:38:35.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-08-30T19:38:37.677 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-08-30T19:38:37.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:38:37.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:38:37.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-30T19:38:37.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-30T19:38:37.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:38:37.749 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-30T19:38:37.749 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:38:37.749 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:38:37.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:38:37.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:38:37.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-30T19:38:37.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-30T19:38:37.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:38:37.752 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-30T19:38:37.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-30T19:38:37.794 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:37.793+0000 7f3dbb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:37.797 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:37.797+0000 7f3dbb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:37.799 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:37.797+0000 7f3dbb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:38.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-30T19:38:38.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:38:38.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:38:38.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:38:38.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:38:38.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:38.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:38:38.176 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:38:38.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:38.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:38.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:38.804 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:38.801+0000 7f3dbb6c67c0 -1 Falling back to public interface 2024-08-30T19:38:39.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:39.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:39.615 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:38:39.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:38:39.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:39.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:40.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:41.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:41.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:41.023 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:38:41.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:38:41.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:41.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:41.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:42.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:42.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:42.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:38:42.461 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:38:42.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:42.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:42.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:43.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:44.244 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:44.241+0000 7f3dbb6c67c0 -1 osd.0 10 log_to_monitors true 2024-08-30T19:38:44.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:45.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:45.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:45.349 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:38:45.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:38:45.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:45.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:46.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:47.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:48.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:38:49.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:38:49.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:38:49.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-30T19:38:49.061 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-08-30T19:38:49.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:38:49.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:38:49.568 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/260048233,v1:127.0.0.1:6803/260048233] [v2:127.0.0.1:6804/260048233,v1:127.0.0.1:6805/260048233] exists,up 38709232-3c93-4c80-b8f2-4944b2e21816 2024-08-30T19:38:49.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:38:49.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:38:49.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:38:49.569 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:38:49.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=148819 2024-08-30T19:38:49.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-08-30T19:38:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-30T19:38:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-30T19:38:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-30T19:38:49.571 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-30T19:38:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-30T19:38:49.572 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-30T19:38:49.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-30T19:38:49.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-30T19:38:49.572 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-08-30T19:38:49.676 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-08-30T19:38:50.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-08-30T19:38:50.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-08-30T19:38:50.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-08-30T19:38:50.125 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-30T19:38:50.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574852 2024-08-30T19:38:50.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574852 2024-08-30T19:38:50.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-55834574852' 2024-08-30T19:38:50.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-08-30T19:38:50.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-55834574852 2024-08-30T19:38:50.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-08-30T19:38:50.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-08-30T19:38:50.291 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-55834574852 2024-08-30T19:38:50.291 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-08-30T19:38:50.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574852 2024-08-30T19:38:50.292 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 55834574852 2024-08-30T19:38:50.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-08-30T19:38:50.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:38:50.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 55834574852 2024-08-30T19:38:50.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-08-30T19:38:51.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-30T19:38:51.689 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-30T19:38:52.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574852 -lt 55834574852 2024-08-30T19:38:52.106 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-30T19:38:52.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:38:52.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:38:52.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-30T19:38:52.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-30T19:38:52.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-30T19:38:52.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-30T19:38:52.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-30T19:38:52.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-30T19:38:52.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-30T19:38:52.662 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-30T19:38:53.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-30T19:38:53.065 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-30T19:38:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-30T19:38:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-30T19:38:53.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-30T19:38:53.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-30T19:38:53.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:38:53.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:38:53.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:38:53.702 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:38:53.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:38:53.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:38:53.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:38:53.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:38:53.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:38:53.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:38:53.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:38:53.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:38:53.706 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:38:53.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:38:53.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:38:53.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:38:53.739 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:38:53.739 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:53.740 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:53.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:38:53.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:38:53.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:38:53.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:38:53.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:38:53.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:38:53.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:38:53.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:38:53.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:38:53.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:38:53.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:38:53.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:38:53.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:38:53.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:38:53.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:38:53.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-30T19:38:53.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:38:53.751 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:38:53.751 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:53.751 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:53.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:38:53.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:38:53.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:38:53.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-30T19:38:53.753 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-30T19:38:53.753 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:53.753 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:53.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97333 2024-08-30T19:38:53.754 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-30T19:38:53.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-30T19:38:53.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-30T19:38:53.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-30T19:38:53.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-08-30T19:38:53.756 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=4843df30-b790-43a7-959b-a739db01f90d --auth-supported=none ' 2024-08-30T19:38:53.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-30T19:38:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-30T19:38:53.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:54.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:54.134 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:54.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-30T19:38:54.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-30T19:38:54.345 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:54.346 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:38:54.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:38:54.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get fsid 2024-08-30T19:38:54.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-30T19:38:54.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-30T19:38:54.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-30T19:38:54.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-30T19:38:54.466 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-30T19:38:54.466 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-30T19:38:54.466 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-30T19:38:54.466 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-30T19:38:54.466 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-30T19:38:54.467 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:54.467 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:54.467 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97333/ceph-mon.a.asok 2024-08-30T19:38:54.467 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-30T19:38:54.467 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97333/ceph-mon.a.asok config get mon_host 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-30T19:38:54.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:55.472 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:55.473 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:55.473 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:38:55.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-30T19:38:55.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4843df30-b790-43a7-959b-a739db01f90d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-30T19:38:55.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-30T19:38:55.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-30T19:38:55.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-30T19:38:55.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=76d05db0-7680-43b0-904c-6eb4160fe3d7 2024-08-30T19:38:55.549 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 76d05db0-7680-43b0-904c-6eb4160fe3d7 2024-08-30T19:38:55.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 76d05db0-7680-43b0-904c-6eb4160fe3d7' 2024-08-30T19:38:55.549 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-30T19:38:55.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPH9JmkZIwIhAA8cCIrkoONKksEfQ/PfCnaQ== 2024-08-30T19:38:55.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPH9JmkZIwIhAA8cCIrkoONKksEfQ/PfCnaQ=="}' 2024-08-30T19:38:55.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 76d05db0-7680-43b0-904c-6eb4160fe3d7 -i td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:38:55.872 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:38:55.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-30T19:38:55.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4843df30-b790-43a7-959b-a739db01f90d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPH9JmkZIwIhAA8cCIrkoONKksEfQ/PfCnaQ== --osd-uuid 76d05db0-7680-43b0-904c-6eb4160fe3d7 2024-08-30T19:38:55.942 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:55.941+0000 7fcfd18227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:55.944 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:55.941+0000 7fcfd18227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:55.945 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:55.945+0000 7fcfd18227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:38:55.946 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:55.945+0000 7fcfd18227c0 -1 bdev(0x562e69f92a80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-30T19:38:55.946 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:38:55.945+0000 7fcfd18227c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-30T19:39:03.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-30T19:39:03.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-30T19:39:03.302 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-08-30T19:39:03.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-30T19:39:03.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-30T19:39:03.872 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-08-30T19:39:03.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-30T19:39:03.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1298dac8-5f99-462e-8df5-9196c323ab8a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a6e21898-d678-45e4-b06e-78b4796303c6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4843df30-b790-43a7-959b-a739db01f90d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97333/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-30T19:39:03.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-30T19:39:03.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-30T19:39:03.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-30T19:39:03.915 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:39:03.912+0000 7f0e72ca37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:39:03.917 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:39:03.916+0000 7f0e72ca37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:39:03.918 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:39:03.916+0000 7f0e72ca37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-30T19:39:04.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-30T19:39:04.332 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-08-30T19:39:04.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:04.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:04.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:05.416 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:39:05.412+0000 7f0e72ca37c0 -1 Falling back to public interface 2024-08-30T19:39:05.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:05.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:05.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-30T19:39:05.768 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-08-30T19:39:05.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:05.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:06.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:07.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:07.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:07.205 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-08-30T19:39:07.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-30T19:39:07.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:07.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:07.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:08.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:08.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:08.650 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-08-30T19:39:08.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-30T19:39:08.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:08.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:09.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:09.662 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:39:09.660+0000 7f0e72ca37c0 -1 osd.0 0 log_to_monitors true 2024-08-30T19:39:10.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:10.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:10.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-30T19:39:10.111 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-08-30T19:39:10.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:10.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:11.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:12.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:12.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:12.266 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-08-30T19:39:12.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-30T19:39:12.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:12.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:13.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-30T19:39:14.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-30T19:39:14.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-30T19:39:14.219 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-08-30T19:39:14.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-30T19:39:14.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-30T19:39:14.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-30T19:39:14.721 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2068821422,v1:127.0.0.1:6803/2068821422] [v2:127.0.0.1:6804/2068821422,v1:127.0.0.1:6805/2068821422] exists,up 76d05db0-7680-43b0-904c-6eb4160fe3d7 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=151550 2024-08-30T19:39:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-08-30T19:39:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-08-30T19:39:19.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-30T19:39:20.336 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-08-30T19:39:20.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-30T19:39:21.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-30T19:39:21.403 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-08-30T19:39:21.404 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-30T19:39:21.404 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_152585 2024-08-30T19:39:21.404 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:39:21.404 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-08-30T19:39:22.403 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 160 144 0.562428 0.5625 0.143619 0.106591 2024-08-30T19:39:23.404 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 305 289 0.564372 0.566406 0.117167 0.106027 2024-08-30T19:39:24.404 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 448 432 0.562415 0.558594 0.0836195 0.10757 2024-08-30T19:39:25.404 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 592 576 0.562417 0.5625 0.084357 0.110442 2024-08-30T19:39:26.404 INFO:tasks.workunit.client.0.smithi079.stdout: 5 15 748 733 0.572577 0.613281 0.0836573 0.108746 2024-08-30T19:39:27.404 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 912 896 0.583251 0.636719 0.101273 0.106297 2024-08-30T19:39:28.405 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 1072 1056 0.5892 0.625 0.10106 0.105538 2024-08-30T19:39:29.405 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 1201 1185 0.578528 0.503906 0.117252 0.107032 2024-08-30T19:39:30.405 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 1313 1297 0.562851 0.4375 0.0754793 0.110052 2024-08-30T19:39:31.405 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 1473 1457 0.569059 0.625 0.100569 0.109013 2024-08-30T19:39:32.405 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 1633 1617 0.574135 0.625 0.100505 0.108002 2024-08-30T19:39:33.405 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 1777 1761 0.573157 0.5625 0.09204 0.108448 2024-08-30T19:39:34.405 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 1937 1921 0.577138 0.625 0.117294 0.107451 2024-08-30T19:39:35.406 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 2096 2080 0.58027 0.621094 0.0925037 0.107563 2024-08-30T19:39:36.408 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 2257 2241 0.583507 0.628906 0.0837863 0.106687 2024-08-30T19:39:37.406 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 2401 2385 0.582187 0.5625 0.117238 0.106787 2024-08-30T19:39:38.406 INFO:tasks.workunit.client.0.smithi079.stdout: 17 15 2545 2530 0.581256 0.566406 0.0920909 0.106994 2024-08-30T19:39:39.406 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 2689 2673 0.579991 0.558594 0.108791 0.107107 2024-08-30T19:39:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 2849 2833 0.582355 0.625 0.0838595 0.106772 2024-08-30T19:39:41.406 INFO:tasks.workunit.client.0.smithi079.stdout:2024-08-30T19:39:41.409337+0000 min lat: 0.0739379 max lat: 0.32548 avg lat: 0.10632 2024-08-30T19:39:41.407 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T19:39:41.407 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 3024 3008 0.587412 0.683594 0.0924769 0.10632 2024-08-30T19:39:42.407 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 3169 3153 0.586407 0.566406 0.108832 0.106228 2024-08-30T19:39:43.407 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 3313 3297 0.585317 0.5625 0.125421 0.106357 2024-08-30T19:39:44.407 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 3473 3457 0.587038 0.625 0.0920389 0.106141 2024-08-30T19:39:45.407 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 3633 3617 0.588616 0.625 0.108938 0.105753 2024-08-30T19:39:46.407 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 3793 3777 0.590068 0.625 0.0818576 0.105734 2024-08-30T19:39:47.407 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 3937 3921 0.589004 0.5625 0.0836951 0.105713 2024-08-30T19:39:48.408 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 4096 4080 0.59019 0.621094 0.10901 0.105806 2024-08-30T19:39:49.408 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 4256 4240 0.591429 0.625 0.0847001 0.105578 2024-08-30T19:39:50.408 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 4417 4401 0.592718 0.628906 0.100446 0.105265 2024-08-30T19:39:51.408 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 4576 4560 0.593661 0.621094 0.116789 0.105218 2024-08-30T19:39:52.408 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1266 2024-08-30T19:39:52.408 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 4576 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.59333 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0480172 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.683594 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.4375 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 151 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 12.2924 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 175 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 112 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.105314 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.02799 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.32548 2024-08-30T19:39:52.409 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0729536 2024-08-30T19:39:52.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-08-30T19:39:57.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-08-30T19:39:57.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-08-30T19:39:57.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-08-30T19:39:57.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-08-30T19:39:57.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4276117504 2024-08-30T19:39:57.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151550 2024-08-30T19:39:57.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-08-30T19:39:58.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151550 2024-08-30T19:39:58.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (151550) - No such process 2024-08-30T19:39:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-08-30T19:39:59.294 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-08-30T19:39:59.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-08-30T19:39:59.346 INFO:tasks.workunit.client.0.smithi079.stdout:allocmap bluestore.allocmap 2024-08-30T19:40:02.084 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:40:02.079+0000 7fc8ce163ac0 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-08-30T19:40:02.084 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:40:02.079+0000 7fc8ce163ac0 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-08-30T19:40:02.085 INFO:tasks.workunit.client.0.smithi079.stderr:2024-08-30T19:40:02.079+0000 7fc8ce163ac0 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-08-30T19:40:02.698 INFO:tasks.workunit.client.0.smithi079.stderr:allocmap failed: (1) Operation not permitted 2024-08-30T19:40:02.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-bluefs-volume-ops 1 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-30T19:40:02.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-30T19:40:02.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-30T19:40:02.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-30T19:40:02.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-30T19:40:02.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-30T19:40:02.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-30T19:40:02.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-30T19:40:02.822 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-30T19:40:02.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-30T19:40:02.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-30T19:40:02.823 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-30T19:40:02.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:40:02.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-30T19:40:02.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-30T19:40:02.824 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T19:40:02.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:40:02.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-08-30T19:40:02.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-08-30T19:40:02.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-08-30T19:40:02.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-08-30T19:40:02.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-30T19:40:02.832 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-30T19:40:02.833 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-30T19:40:02.833 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97333 2024-08-30T19:40:02.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97333 2024-08-30T19:40:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T19:40:02.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-30T19:40:02.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-30T19:40:02.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-08-30T19:40:02.835 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-08-30T19:40:02.836 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-30T19:40:03.185 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-08-30T19:40:03.358 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0216ea22e73843febeac4fcb671138b9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-08-30T19:40:03.361 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-30T19:40:03.368 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-30T19:40:03.369 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-30T19:40:03.431 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-30T19:40:03.438 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-30T19:40:03.439 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T19:40:03.451 INFO:teuthology.orchestra.run.smithi079.stdout: remote refid st t when poll reach delay offset jitter 2024-08-30T19:40:03.451 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================== 2024-08-30T19:40:03.451 INFO:teuthology.orchestra.run.smithi079.stdout:*hv01.front.sepi 67.205.162.81 3 u 22 64 377 0.101 -4.355 0.568 2024-08-30T19:40:03.451 INFO:teuthology.orchestra.run.smithi079.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 64 377 0.107 -3.377 0.610 2024-08-30T19:40:03.451 INFO:teuthology.orchestra.run.smithi079.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 377 0.107 -4.726 0.557 2024-08-30T19:40:03.452 INFO:teuthology.orchestra.run.smithi079.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-08-30T19:40:03.452 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-30T19:40:03.507 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-30T19:40:03.507 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-30T19:40:03.540 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-30T19:40:03.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-30T19:40:03.607 INFO:teuthology.task.internal:Duration was 2958.072538 seconds 2024-08-30T19:40:03.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-30T19:40:03.641 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-30T19:40:03.641 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-30T19:40:03.685 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-30T19:40:03.685 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-08-30T19:40:03.686 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-30T19:40:03.739 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-30T19:40:03.739 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-30T19:40:03.817 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-30T19:40:03.817 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-30T19:40:03.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-30T19:40:03.920 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-30T19:40:03.920 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-30T19:40:03.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-30T19:40:03.961 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:40:03.983 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-08-30T19:40:04.006 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-30T19:40:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T19:40:04.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-30T19:40:04.062 INFO:teuthology.task.internal:Transferring archived files... 2024-08-30T19:40:04.064 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-08-30_15:04:49-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7881988/remote/smithi079 2024-08-30T19:40:04.065 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-30T19:40:07.003 INFO:teuthology.task.internal:Removing archive directory... 2024-08-30T19:40:07.004 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-30T19:40:07.071 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-30T19:40:07.080 INFO:teuthology.task.internal:Not uploading archives. 2024-08-30T19:40:07.081 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-30T19:40:07.154 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-30T19:40:07.155 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-30T19:40:07.162 INFO:teuthology.orchestra.run.smithi079.stdout: 658466 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 30 19:40 /home/ubuntu/cephtest 2024-08-30T19:40:07.162 INFO:teuthology.orchestra.run.smithi079.stdout: 788823 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 30 19:00 /home/ubuntu/cephtest/mnt.0 2024-08-30T19:40:07.162 INFO:teuthology.orchestra.run.smithi079.stdout: 789008 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 30 19:27 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-30T19:40:07.162 INFO:teuthology.orchestra.run.smithi079.stdout: 791840 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 30 19:27 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-30T19:40:07.163 INFO:teuthology.orchestra.run.smithi079.stdout: 806427 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 30 19:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-08-30T19:40:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T19:40:07.163 INFO:teuthology.orchestra.run.smithi079.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-30T19:40:07.164 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-30T19:40:07.168 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-30T19:40:07.183 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-30T19:40:07.217 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi079 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-30T19:40:07.220 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 2958.072538137436 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0216ea22e73843febeac4fcb671138b9 status: fail success: false 2024-08-30T19:40:07.220 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-30T19:40:07.308 INFO:teuthology.run:FAIL