2024-09-04T19:02:09.199 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-04T19:02:09.212 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-04T19:02:09.295 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-09-04_18:54:56-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7889511 branch: wip-yuri8-testing-2024-08-28-1632-squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7889511' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: lflores-2024-09-04_18:54:56-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri8-testing-2024-08-28-1632-squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 workunit: branch: wip-bz-2305110-squid sha1: 055b029e365e266078e29644ed3ff217dfe73d04 owner: scheduled_lflores@folio02 priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2672 sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bz-2305110-squid suite_path: /home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 055b029e365e266078e29644ed3ff217dfe73d04 targets: smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC7Ea0wR6o0d5vQ2fdJ4EKzEU9YuzOf98L/0n3hC8s0RnHfDGFeqiz/8SRUZslJrl6e1/jA6Wr7iYXC9X6N0dvk= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-04_18:54:56 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-04T19:02:09.295 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa; will attempt to use it 2024-09-04T19:02:09.296 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks 2024-09-04T19:02:09.296 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-04T19:02:09.297 INFO:teuthology.task.internal:Checking packages... 2024-09-04T19:02:09.317 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1ef864504b8875c83ee6c2c5fedc13315bebf7f5' 2024-09-04T19:02:09.317 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-04T19:02:09.317 INFO:teuthology.packaging:ref: None 2024-09-04T19:02:09.317 INFO:teuthology.packaging:tag: None 2024-09-04T19:02:09.317 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-08-28-1632-squid 2024-09-04T19:02:09.317 INFO:teuthology.packaging:sha1: 1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-09-04T19:02:09.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri8-testing-2024-08-28-1632-squid 2024-09-04T19:02:09.465 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-235-g1ef86450-1jammy 2024-09-04T19:02:09.466 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-04T19:02:09.473 INFO:teuthology.task.internal:no buildpackages task found 2024-09-04T19:02:09.473 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-04T19:02:09.627 INFO:teuthology.task.internal:Saving configuration 2024-09-04T19:02:09.636 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-04T19:02:09.757 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-04T19:02:09.783 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-09-04_18:54:56-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7889511', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-04 18:56:47.372281', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPV5R6WfHvztuhhSgTvrRfcN5V7gYl7KgNdgcVh0hm5c'} 2024-09-04T19:02:09.783 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-04T19:02:09.945 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-04T19:02:09.946 INFO:teuthology.run_tasks:Running task console_log... 2024-09-04T19:02:10.051 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f707743fbe0>, signals=[15]) 2024-09-04T19:02:10.051 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-04T19:02:10.122 INFO:teuthology.task.internal:Opening connections... 2024-09-04T19:02:10.122 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-04T19:02:10.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T19:02:10.204 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-04T19:02:10.210 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-04T19:02:10.305 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-04T19:02:10.305 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-04T19:02:10.350 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-04T19:02:10.350 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Ubuntu" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="22.04" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_CODENAME=jammy 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:ID=ubuntu 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE=debian 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-04T19:02:10.351 INFO:teuthology.orchestra.run.smithi174.stdout:UBUNTU_CODENAME=jammy 2024-09-04T19:02:10.352 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-04T19:02:10.376 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-04T19:02:10.384 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-04T19:02:10.444 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-04T19:02:10.444 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-04T19:02:10.449 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-04T19:02:10.535 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-04T19:02:10.536 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-04T19:02:10.567 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-04T19:02:10.664 INFO:teuthology.run_tasks:Running task kernel... 2024-09-04T19:02:10.720 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-04T19:02:10.720 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-04T19:02:10.721 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-04T19:02:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T19:02:10.726 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-04T19:02:10.775 INFO:teuthology.orchestra.run.smithi174.stdout:5.15.0-56-generic 2024-09-04T19:02:10.775 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.15.0-56-generic 2024-09-04T19:02:10.775 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-04T19:02:10.906 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-04T19:02:11.043 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T19:02:11.045 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-04T19:02:11.045 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-04T19:02:11.121 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-04T19:02:11.278 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-04T19:02:11.346 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-04T19:02:11.353 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-04T19:02:11.383 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-04T19:02:11.386 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-04T19:02:11.386 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-04T19:02:11.404 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-04T19:02:11.404 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-04T19:02:11.407 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-04T19:02:11.407 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-04T19:02:11.411 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-04T19:02:11.414 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-04T19:02:11.418 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-04T19:02:11.438 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-04T19:02:11.438 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-04T19:02:11.439 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-04T19:02:11.439 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-04T19:02:11.439 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-04T19:02:11.449 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-04T19:02:11.453 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-04T19:02:11.453 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-04T19:02:11.469 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-04T19:02:11.469 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-04T19:02:11.471 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-04T19:02:11.471 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,121 kB] 2024-09-04T19:02:11.478 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [730 kB] 2024-09-04T19:02:11.483 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [260 kB] 2024-09-04T19:02:11.485 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-04T19:02:11.486 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-04T19:02:11.486 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-04T19:02:11.502 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-04T19:02:11.502 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-04T19:02:11.502 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-04T19:02:11.503 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-04T19:02:11.503 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-04T19:02:11.504 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-04T19:02:11.518 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-04T19:02:11.519 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-04T19:02:11.535 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-04T19:02:11.535 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-04T19:02:17.568 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 15.8 MB in 2s (7,550 kB/s) 2024-09-04T19:02:18.975 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:02:18.993 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T19:02:19.097 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:02:19.306 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-04T19:02:19.307 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-04T19:02:19.495 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T19:02:19.495 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T19:02:19.495 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T19:02:19.496 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T19:02:19.496 INFO:teuthology.orchestra.run.smithi174.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-04T19:02:19.496 INFO:teuthology.orchestra.run.smithi174.stdout: python2.7-minimal 2024-09-04T19:02:19.496 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T19:02:19.497 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-04T19:02:19.497 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T19:02:19.497 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-04T19:02:19.497 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-04T19:02:19.499 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-04T19:02:19.499 INFO:teuthology.orchestra.run.smithi174.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-04T19:02:19.499 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-04T19:02:19.499 INFO:teuthology.orchestra.run.smithi174.stdout: thermald 2024-09-04T19:02:19.555 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-04T19:02:19.555 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-04T19:02:19.555 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-04T19:02:19.555 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-04T19:02:19.556 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-09-04T19:02:19.557 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-04T19:02:19.736 INFO:teuthology.orchestra.run.smithi174.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T19:02:19.736 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 113 MB of archives. 2024-09-04T19:02:19.736 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-04T19:02:19.736 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-04T19:02:21.148 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-04T19:02:21.496 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-04T19:02:23.676 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-04T19:02:23.676 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-04T19:02:23.677 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-04T19:02:24.022 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-04T19:02:24.103 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-04T19:02:24.500 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 113 MB in 5s (25.0 MB/s) 2024-09-04T19:02:24.801 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-04T19:02:26.688 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-04T19:02:26.692 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T19:02:26.747 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:28.922 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-04T19:02:28.943 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T19:02:29.030 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:29.491 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-04T19:02:29.511 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T19:02:29.558 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:36.087 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T19:02:36.213 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T19:02:36.538 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T19:02:36.666 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T19:02:36.961 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-04T19:02:36.983 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-04T19:02:37.036 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T19:02:42.906 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-04T19:02:42.942 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-04T19:02:42.972 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:45.524 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-04T19:02:45.647 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-04T19:02:45.997 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-04T19:02:46.118 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:46.244 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-04T19:02:46.434 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:48.555 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-04T19:02:48.555 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-04T19:02:48.555 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-04T19:02:48.556 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-04T19:02:48.716 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:50.126 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:51.527 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-04T19:02:51.653 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-04T19:02:51.779 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-04T19:02:51.845 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-04T19:02:51.846 INFO:teuthology.orchestra.run.smithi174.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-04T19:03:05.973 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-04T19:03:05.973 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub' 2024-09-04T19:03:05.989 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T19:03:06.008 INFO:teuthology.orchestra.run.smithi174.stdout:Generating grub configuration file ... 2024-09-04T19:03:06.249 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T19:03:06.276 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T19:03:06.608 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T19:03:06.614 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T19:03:06.683 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T19:03:06.689 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T19:03:06.861 INFO:teuthology.orchestra.run.smithi174.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T19:03:06.861 INFO:teuthology.orchestra.run.smithi174.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T19:03:06.861 INFO:teuthology.orchestra.run.smithi174.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T19:03:06.906 INFO:teuthology.orchestra.run.smithi174.stdout:done 2024-09-04T19:03:07.392 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-04T19:03:07.412 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-04T19:03:07.413 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-04T19:03:07.413 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-04T19:03:07.413 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 21 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.15.0.119.119 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T19:03:07.416 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-04T19:03:07.417 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-04T19:03:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-04T19:03:07.417 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-04T19:03:07.417 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-04T19:03:07.417 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-04T19:03:07.417 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-04T19:03:07.417 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-04T19:03:07.417 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-09-04T19:03:07.418 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-04T19:03:07.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-04T19:03:07.741 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T19:03:07.742 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-04T19:03:07.743 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-04T19:03:07.760 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T19:03:09.140 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:03:09.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-04T19:03:09.213 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:03:09.399 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-04T19:03:09.400 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-04T19:03:09.568 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-04T19:03:09.569 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T19:03:09.569 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T19:03:09.569 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T19:03:09.569 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T19:03:09.569 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T19:03:09.570 INFO:teuthology.orchestra.run.smithi174.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T19:03:09.570 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T19:03:09.608 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-04T19:03:09.610 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-04T19:03:09.669 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 21 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.15.0.119.119 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-04T19:03:09.670 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-04T19:03:09.671 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-04T19:03:09.671 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-04T19:03:09.671 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-04T19:03:09.715 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.1AGBsavcRQ 2024-09-04T19:03:09.715 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1AGBsavcRQ 2024-09-04T19:03:09.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.1AGBsavcRQ 2024-09-04T19:03:09.883 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.1AGBsavcRQ is 10KB 2024-09-04T19:03:09.895 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.1AGBsavcRQ 2024-09-04T19:03:09.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-04T19:03:09.954 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:03:09.954 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-04T19:03:09.954 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-04T19:03:10.022 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-04T19:03:10.022 DEBUG:teuthology.orchestra.run.smithi174:> sudo update-grub 2024-09-04T19:03:11.197 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub' 2024-09-04T19:03:11.198 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-04T19:03:11.200 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-09-04T19:03:11.398 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-04T19:03:11.404 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-04T19:03:11.638 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-04T19:03:11.643 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-04T19:03:11.685 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-04T19:03:11.690 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-04T19:03:11.813 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-04T19:03:11.813 INFO:teuthology.orchestra.run.smithi174.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-04T19:03:11.813 INFO:teuthology.orchestra.run.smithi174.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-04T19:03:11.838 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-09-04T19:03:11.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-09-04T19:03:41.877 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-04T19:03:41.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-04T19:03:41.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T19:04:41.938 DEBUG:teuthology.orchestra.remote:timed out 2024-09-04T19:04:50.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-04T19:04:50.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T19:04:51.301 ERROR:teuthology.orchestra.connection:Error authenticating with smithi174.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-09-04T19:04:55.387 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-04T19:04:56.035 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-04T19:04:56.035 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-04T19:04:56.035 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-04T19:04:56.081 INFO:teuthology.orchestra.run.smithi174.stdout:5.15.0-119-generic 2024-09-04T19:04:56.081 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-04T19:04:56.081 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-04T19:04:56.081 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-04T19:04:57.082 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-04T19:04:57.083 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-04T19:04:57.180 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-09-04T19:04:57.206 DEBUG:teuthology.parallel:result is None 2024-09-04T19:04:57.207 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-04T19:04:57.217 INFO:teuthology.task.internal:Creating test directory... 2024-09-04T19:04:57.217 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-04T19:04:57.223 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-04T19:04:57.370 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-04T19:04:57.499 INFO:teuthology.task.internal:Creating archive directory... 2024-09-04T19:04:57.499 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-04T19:04:57.520 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-04T19:04:57.602 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-04T19:04:57.603 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-04T19:04:57.633 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:04:57.641 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:04:57.643 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-04T19:04:57.781 INFO:teuthology.task.internal:Configuring sudo... 2024-09-04T19:04:57.781 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-04T19:04:57.797 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-04T19:04:57.879 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-04T19:04:57.880 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-04T19:04:57.886 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-04T19:04:57.933 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-04T19:04:57.981 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:04:57.981 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-04T19:04:58.041 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-04T19:04:58.119 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-04T19:04:58.128 INFO:teuthology.task.internal:Starting timer... 2024-09-04T19:04:58.128 INFO:teuthology.run_tasks:Running task pcp... 2024-09-04T19:04:58.186 INFO:teuthology.run_tasks:Running task selinux... 2024-09-04T19:04:58.255 DEBUG:teuthology.task.selinux:Excluding smithi174: OS 'ubuntu' does not support SELinux 2024-09-04T19:04:58.255 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-04T19:04:58.256 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-04T19:04:58.256 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-04T19:04:58.256 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-04T19:04:58.575 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-04T19:04:58.587 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-04T19:04:58.588 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-04T19:12:23.613 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-09-04T19:12:23.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-04T19:12:23.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-04T19:12:23.698 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-04T19:12:23.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-04T19:12:23.777 INFO:teuthology.run_tasks:Running task clock... 2024-09-04T19:12:23.788 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-04T19:12:23.788 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-04T19:12:23.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T19:12:23.849 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-04T19:12:23.849 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Command line: ntpd -gq 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: ---------------------------------------------------- 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: ntp-4 is maintained by Network Time Foundation, 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: corporation. Support and training for ntp-4 are 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: available at https://www.nwtime.org/support 2024-09-04T19:12:23.850 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: ---------------------------------------------------- 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: proto: precision = 0.043 usec (-24) 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: basedate set to 2022-02-04 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: gps base set to 2022-02-06 (week 2196) 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-04T19:12:23.851 INFO:teuthology.orchestra.run.smithi174.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-04T19:12:23.852 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: restrict ::: KOD does nothing without LIMITED. 2024-09-04T19:12:23.852 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen and drop on 0 v6wildcard [::]:123 2024-09-04T19:12:23.852 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-04T19:12:23.853 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen normally on 2 lo 127.0.0.1:123 2024-09-04T19:12:23.853 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen normally on 3 enp3s0f1 172.21.15.174:123 2024-09-04T19:12:23.853 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen normally on 4 lo [::1]:123 2024-09-04T19:12:23.853 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:771b%5]:123 2024-09-04T19:12:23.853 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:23 ntpd[18486]: Listening on routing socket on fd #22 for interface updates 2024-09-04T19:12:24.850 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:24 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:24.850 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:24 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:26.850 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:26.850 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:26.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:26.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:26.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:26.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:26 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:28.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:28 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:30.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:30.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:30.853 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:30.853 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:30.853 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:30 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:32.850 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240904: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-04T19:12:32.851 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240904: Permission denied 2024-09-04T19:12:32.852 INFO:teuthology.orchestra.run.smithi174.stdout: 4 Sep 19:12:32 ntpd[18486]: ntpd: time slew -0.000485 s 2024-09-04T19:12:32.852 INFO:teuthology.orchestra.run.smithi174.stdout:ntpd: time slew -0.000485s 2024-09-04T19:12:32.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-04T19:12:32.852 INFO:teuthology.orchestra.run.smithi174.stderr: 4 Sep 19:12:32 ntpd[18486]: can't open /var/log/ntpstats/loopstats.20240904: Permission denied 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T19:12:32.913 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-04T19:12:32.914 INFO:teuthology.run_tasks:Running task install... 2024-09-04T19:12:32.923 DEBUG:teuthology.task.install:project ceph 2024-09-04T19:12:32.924 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1ef864504b8875c83ee6c2c5fedc13315bebf7f5'}} 2024-09-04T19:12:32.924 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1ef864504b8875c83ee6c2c5fedc13315bebf7f5'} 2024-09-04T19:12:32.924 INFO:teuthology.task.install:Using flavor: default 2024-09-04T19:12:32.929 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-04T19:12:32.929 INFO:teuthology.task.install:extra packages: [] 2024-09-04T19:12:32.930 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-key list | grep Ceph 2024-09-04T19:12:33.017 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-04T19:12:33.047 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-04T19:12:33.047 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph.com (release key) 2024-09-04T19:12:33.047 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-04T19:12:33.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-09-04T19:12:33.190 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default/ 2024-09-04T19:12:33.190 INFO:teuthology.task.install.deb:Package version is 19.1.1-235-g1ef86450-1jammy 2024-09-04T19:12:33.301 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:12:33.301 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-04T19:12:33.315 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-04T19:12:33.543 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy InRelease 2024-09-04T19:12:33.565 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-04T19:12:33.589 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-04T19:12:33.618 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-04T19:12:33.619 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-04T19:12:33.633 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-04T19:12:33.674 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,405 B] 2024-09-04T19:12:33.690 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-04T19:12:33.771 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-04T19:12:33.991 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 28.9 kB in 0s (58.9 kB/s) 2024-09-04T19:12:35.005 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:12:35.021 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-235-g1ef86450-1jammy cephadm=19.1.1-235-g1ef86450-1jammy ceph-mds=19.1.1-235-g1ef86450-1jammy ceph-mgr=19.1.1-235-g1ef86450-1jammy ceph-common=19.1.1-235-g1ef86450-1jammy ceph-fuse=19.1.1-235-g1ef86450-1jammy ceph-test=19.1.1-235-g1ef86450-1jammy ceph-volume=19.1.1-235-g1ef86450-1jammy radosgw=19.1.1-235-g1ef86450-1jammy python3-rados=19.1.1-235-g1ef86450-1jammy python3-rgw=19.1.1-235-g1ef86450-1jammy python3-cephfs=19.1.1-235-g1ef86450-1jammy python3-rbd=19.1.1-235-g1ef86450-1jammy libcephfs2=19.1.1-235-g1ef86450-1jammy libcephfs-dev=19.1.1-235-g1ef86450-1jammy librados2=19.1.1-235-g1ef86450-1jammy librbd1=19.1.1-235-g1ef86450-1jammy rbd-fuse=19.1.1-235-g1ef86450-1jammy 2024-09-04T19:12:35.077 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-04T19:12:35.269 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-04T19:12:35.270 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-04T19:12:35.459 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-04T19:12:35.460 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-04T19:12:35.461 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-04T19:12:35.461 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-04T19:12:35.461 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-04T19:12:35.463 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-04T19:12:35.464 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet zip 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-04T19:12:35.465 INFO:teuthology.orchestra.run.smithi174.stdout: btrfs-tools 2024-09-04T19:12:35.567 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-04T19:12:35.567 INFO:teuthology.orchestra.run.smithi174.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-04T19:12:35.568 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-04T19:12:35.568 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-04T19:12:35.568 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-04T19:12:35.568 INFO:teuthology.orchestra.run.smithi174.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-04T19:12:35.568 INFO:teuthology.orchestra.run.smithi174.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-04T19:12:35.569 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-04T19:12:35.570 INFO:teuthology.orchestra.run.smithi174.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-04T19:12:35.628 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-04T19:12:35.628 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 181 MB of archives. 2024-09-04T19:12:35.628 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-04T19:12:35.629 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-04T19:12:35.716 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-04T19:12:35.718 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-04T19:12:35.735 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-04T19:12:35.760 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-235-g1ef86450-1jammy [3,590 kB] 2024-09-04T19:12:35.793 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-04T19:12:35.798 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-04T19:12:35.810 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-04T19:12:35.814 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-04T19:12:35.815 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-04T19:12:35.816 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-04T19:12:35.824 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-04T19:12:35.826 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-04T19:12:35.829 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-04T19:12:35.830 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-04T19:12:35.831 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-04T19:12:35.833 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-04T19:12:35.847 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-04T19:12:35.849 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-04T19:12:35.850 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-04T19:12:35.850 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-04T19:12:35.851 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-04T19:12:35.852 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-04T19:12:35.852 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-04T19:12:35.853 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-04T19:12:35.864 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-04T19:12:35.865 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-04T19:12:35.882 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-04T19:12:35.884 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-04T19:12:35.885 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-04T19:12:35.885 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-04T19:12:35.886 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-04T19:12:35.887 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-04T19:12:35.887 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-04T19:12:35.888 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-04T19:12:35.899 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-04T19:12:35.900 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-04T19:12:35.916 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-04T19:12:35.921 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-04T19:12:35.921 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-04T19:12:35.922 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-04T19:12:35.923 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-04T19:12:35.924 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-04T19:12:35.926 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-04T19:12:35.927 INFO:teuthology.orchestra.run.smithi174.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-04T19:12:35.933 INFO:teuthology.orchestra.run.smithi174.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-04T19:12:35.936 INFO:teuthology.orchestra.run.smithi174.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-04T19:12:35.949 INFO:teuthology.orchestra.run.smithi174.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-04T19:12:35.950 INFO:teuthology.orchestra.run.smithi174.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-04T19:12:35.951 INFO:teuthology.orchestra.run.smithi174.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-04T19:12:35.953 INFO:teuthology.orchestra.run.smithi174.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-04T19:12:35.972 INFO:teuthology.orchestra.run.smithi174.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-235-g1ef86450-1jammy [3,240 kB] 2024-09-04T19:12:36.006 INFO:teuthology.orchestra.run.smithi174.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-04T19:12:36.006 INFO:teuthology.orchestra.run.smithi174.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-04T19:12:36.008 INFO:teuthology.orchestra.run.smithi174.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-04T19:12:36.008 INFO:teuthology.orchestra.run.smithi174.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-04T19:12:36.009 INFO:teuthology.orchestra.run.smithi174.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-04T19:12:36.026 INFO:teuthology.orchestra.run.smithi174.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-04T19:12:36.028 INFO:teuthology.orchestra.run.smithi174.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-04T19:12:36.028 INFO:teuthology.orchestra.run.smithi174.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-04T19:12:36.043 INFO:teuthology.orchestra.run.smithi174.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-04T19:12:36.043 INFO:teuthology.orchestra.run.smithi174.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-04T19:12:36.043 INFO:teuthology.orchestra.run.smithi174.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-04T19:12:36.044 INFO:teuthology.orchestra.run.smithi174.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-04T19:12:36.044 INFO:teuthology.orchestra.run.smithi174.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-04T19:12:36.045 INFO:teuthology.orchestra.run.smithi174.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-04T19:12:36.058 INFO:teuthology.orchestra.run.smithi174.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-04T19:12:36.059 INFO:teuthology.orchestra.run.smithi174.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-04T19:12:36.062 INFO:teuthology.orchestra.run.smithi174.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-04T19:12:36.075 INFO:teuthology.orchestra.run.smithi174.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-04T19:12:36.076 INFO:teuthology.orchestra.run.smithi174.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-04T19:12:36.077 INFO:teuthology.orchestra.run.smithi174.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-04T19:12:36.080 INFO:teuthology.orchestra.run.smithi174.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-04T19:12:36.082 INFO:teuthology.orchestra.run.smithi174.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-04T19:12:36.083 INFO:teuthology.orchestra.run.smithi174.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-04T19:12:36.083 INFO:teuthology.orchestra.run.smithi174.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-04T19:12:36.092 INFO:teuthology.orchestra.run.smithi174.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-04T19:12:36.093 INFO:teuthology.orchestra.run.smithi174.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-04T19:12:36.095 INFO:teuthology.orchestra.run.smithi174.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-04T19:12:36.109 INFO:teuthology.orchestra.run.smithi174.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-04T19:12:36.110 INFO:teuthology.orchestra.run.smithi174.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-04T19:12:36.110 INFO:teuthology.orchestra.run.smithi174.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-04T19:12:36.112 INFO:teuthology.orchestra.run.smithi174.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-04T19:12:36.112 INFO:teuthology.orchestra.run.smithi174.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-04T19:12:36.118 INFO:teuthology.orchestra.run.smithi174.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-04T19:12:36.119 INFO:teuthology.orchestra.run.smithi174.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-04T19:12:36.126 INFO:teuthology.orchestra.run.smithi174.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-04T19:12:36.128 INFO:teuthology.orchestra.run.smithi174.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-04T19:12:36.129 INFO:teuthology.orchestra.run.smithi174.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-04T19:12:36.135 INFO:teuthology.orchestra.run.smithi174.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-235-g1ef86450-1jammy [719 kB] 2024-09-04T19:12:36.171 INFO:teuthology.orchestra.run.smithi174.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-235-g1ef86450-1jammy [357 kB] 2024-09-04T19:12:36.220 INFO:teuthology.orchestra.run.smithi174.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-235-g1ef86450-1jammy [32.8 kB] 2024-09-04T19:12:36.247 INFO:teuthology.orchestra.run.smithi174.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-235-g1ef86450-1jammy [180 kB] 2024-09-04T19:12:36.297 INFO:teuthology.orchestra.run.smithi174.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-235-g1ef86450-1jammy [67.6 kB] 2024-09-04T19:12:36.323 INFO:teuthology.orchestra.run.smithi174.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-235-g1ef86450-1jammy [333 kB] 2024-09-04T19:12:36.348 INFO:teuthology.orchestra.run.smithi174.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-235-g1ef86450-1jammy [6,874 kB] 2024-09-04T19:12:36.516 INFO:teuthology.orchestra.run.smithi174.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-235-g1ef86450-1jammy [112 kB] 2024-09-04T19:12:36.536 INFO:teuthology.orchestra.run.smithi174.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-235-g1ef86450-1jammy [472 kB] 2024-09-04T19:12:36.568 INFO:teuthology.orchestra.run.smithi174.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-235-g1ef86450-1jammy [25.9 MB] 2024-09-04T19:12:37.431 INFO:teuthology.orchestra.run.smithi174.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-235-g1ef86450-1jammy [5,144 kB] 2024-09-04T19:12:37.510 INFO:teuthology.orchestra.run.smithi174.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-235-g1ef86450-1jammy [245 kB] 2024-09-04T19:12:37.512 INFO:teuthology.orchestra.run.smithi174.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-235-g1ef86450-1jammy [124 kB] 2024-09-04T19:12:37.514 INFO:teuthology.orchestra.run.smithi174.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-235-g1ef86450-1jammy [1,555 kB] 2024-09-04T19:12:37.534 INFO:teuthology.orchestra.run.smithi174.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-235-g1ef86450-1jammy [6,127 kB] 2024-09-04T19:12:37.743 INFO:teuthology.orchestra.run.smithi174.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-235-g1ef86450-1jammy [22.8 MB] 2024-09-04T19:12:38.688 INFO:teuthology.orchestra.run.smithi174.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-235-g1ef86450-1jammy [14.1 kB] 2024-09-04T19:12:38.688 INFO:teuthology.orchestra.run.smithi174.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-235-g1ef86450-1jammy [877 kB] 2024-09-04T19:12:38.705 INFO:teuthology.orchestra.run.smithi174.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-235-g1ef86450-1jammy [2,462 kB] 2024-09-04T19:12:38.748 INFO:teuthology.orchestra.run.smithi174.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-235-g1ef86450-1jammy [794 kB] 2024-09-04T19:12:38.758 INFO:teuthology.orchestra.run.smithi174.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-235-g1ef86450-1jammy [152 kB] 2024-09-04T19:12:38.760 INFO:teuthology.orchestra.run.smithi174.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-235-g1ef86450-1jammy [2,323 kB] 2024-09-04T19:12:38.782 INFO:teuthology.orchestra.run.smithi174.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-235-g1ef86450-1jammy [8,625 kB] 2024-09-04T19:12:38.989 INFO:teuthology.orchestra.run.smithi174.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-235-g1ef86450-1jammy [14.2 kB] 2024-09-04T19:12:39.008 INFO:teuthology.orchestra.run.smithi174.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-235-g1ef86450-1jammy [51.3 MB] 2024-09-04T19:12:40.896 INFO:teuthology.orchestra.run.smithi174.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-235-g1ef86450-1jammy [130 kB] 2024-09-04T19:12:40.899 INFO:teuthology.orchestra.run.smithi174.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-235-g1ef86450-1jammy [39.0 kB] 2024-09-04T19:12:40.900 INFO:teuthology.orchestra.run.smithi174.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-235-g1ef86450-1jammy [13.6 MB] 2024-09-04T19:12:41.196 INFO:teuthology.orchestra.run.smithi174.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-08-28-1632-squid/1ef864504b8875c83ee6c2c5fedc13315bebf7f5/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-235-g1ef86450-1jammy [91.6 kB] 2024-09-04T19:12:41.772 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 181 MB in 6s (32.2 MB/s) 2024-09-04T19:12:41.906 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-04T19:12:42.004 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-04T19:12:42.010 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-04T19:12:42.088 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T19:12:42.416 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-04T19:12:42.442 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-04T19:12:42.474 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T19:12:42.802 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-04T19:12:42.829 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-04T19:12:42.897 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T19:12:43.430 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-04T19:12:43.457 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T19:12:43.505 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:12:43.900 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-04T19:12:43.926 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T19:12:43.966 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:12:44.278 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-04T19:12:44.304 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-04T19:12:44.344 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:12:44.698 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-04T19:12:44.724 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-04T19:12:44.764 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T19:12:45.083 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librados2. 2024-09-04T19:12:45.110 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../007-librados2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:45.150 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librados2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:45.553 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librbd1. 2024-09-04T19:12:45.579 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../008-librbd1_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:45.611 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librbd1 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:46.047 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs2. 2024-09-04T19:12:46.073 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:46.114 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:46.391 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rados. 2024-09-04T19:12:46.417 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../010-python3-rados_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:46.457 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rados (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:46.811 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-04T19:12:46.836 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:12:46.877 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-argparse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:47.138 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cephfs. 2024-09-04T19:12:47.164 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:47.204 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cephfs (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:47.482 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-04T19:12:47.497 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:12:47.532 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:47.827 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-04T19:12:47.853 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-04T19:12:47.893 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T19:12:48.204 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-prettytable. 2024-09-04T19:12:48.230 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-04T19:12:48.270 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-04T19:12:48.523 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rbd. 2024-09-04T19:12:48.538 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:48.581 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rbd (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:48.901 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-04T19:12:48.927 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-04T19:12:48.967 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T19:12:49.220 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-04T19:12:49.235 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-04T19:12:49.278 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T19:12:49.573 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-04T19:12:49.598 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-04T19:12:49.639 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T19:12:49.959 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua5.1. 2024-09-04T19:12:49.985 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-04T19:12:50.025 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-04T19:12:50.337 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-any. 2024-09-04T19:12:50.361 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-04T19:12:50.403 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-04T19:12:50.664 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package zip. 2024-09-04T19:12:50.691 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-04T19:12:50.731 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking zip (3.0-12build2) ... 2024-09-04T19:12:50.992 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package unzip. 2024-09-04T19:12:51.007 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-04T19:12:51.050 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-04T19:12:51.454 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package luarocks. 2024-09-04T19:12:51.481 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-04T19:12:51.520 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-04T19:12:51.845 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librgw2. 2024-09-04T19:12:51.860 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../025-librgw2_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:51.903 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librgw2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:52.716 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rgw. 2024-09-04T19:12:52.729 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:52.772 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rgw (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:53.108 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-04T19:12:53.134 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-04T19:12:53.174 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T19:12:53.461 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libradosstriper1. 2024-09-04T19:12:53.487 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:53.527 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libradosstriper1 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:53.906 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-common. 2024-09-04T19:12:53.932 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../029-ceph-common_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:53.972 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:55.336 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-base. 2024-09-04T19:12:55.362 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../030-ceph-base_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:12:55.411 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-base (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:12:55.981 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-04T19:12:56.008 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-04T19:12:56.048 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-04T19:12:56.350 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cheroot. 2024-09-04T19:12:56.377 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-04T19:12:56.417 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T19:12:56.720 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-04T19:12:56.746 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-04T19:12:56.786 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-04T19:12:57.056 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-04T19:12:57.083 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-04T19:12:57.122 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-04T19:12:57.425 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-04T19:12:57.451 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-04T19:12:57.491 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-04T19:12:57.803 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempora. 2024-09-04T19:12:57.829 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-04T19:12:57.869 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-04T19:12:58.147 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-portend. 2024-09-04T19:12:58.174 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-04T19:12:58.239 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-04T19:12:58.525 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-04T19:12:58.551 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-04T19:12:58.591 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-04T19:12:58.878 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-04T19:12:58.904 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-04T19:12:58.944 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-04T19:12:59.264 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-natsort. 2024-09-04T19:12:59.291 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-04T19:12:59.331 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-04T19:12:59.642 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-logutils. 2024-09-04T19:12:59.668 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-04T19:12:59.708 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-04T19:13:00.020 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-04T19:13:00.046 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-04T19:13:00.086 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-04T19:13:00.364 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-mako. 2024-09-04T19:13:00.390 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-04T19:13:00.430 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T19:13:00.733 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-04T19:13:00.749 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-04T19:13:00.783 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-04T19:13:01.080 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-04T19:13:01.106 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-04T19:13:01.146 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-04T19:13:01.424 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webob. 2024-09-04T19:13:01.450 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-04T19:13:01.490 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T19:13:01.810 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-waitress. 2024-09-04T19:13:01.836 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-04T19:13:01.880 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T19:13:02.188 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempita. 2024-09-04T19:13:02.214 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-04T19:13:02.254 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T19:13:02.557 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-paste. 2024-09-04T19:13:02.583 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-04T19:13:02.623 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T19:13:03.152 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-04T19:13:03.178 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-04T19:13:03.218 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T19:13:03.463 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-04T19:13:03.489 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-04T19:13:03.529 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-04T19:13:03.857 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webtest. 2024-09-04T19:13:03.883 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-04T19:13:03.923 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-04T19:13:04.218 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pecan. 2024-09-04T19:13:04.244 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-04T19:13:04.284 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T19:13:04.648 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-certifi. 2024-09-04T19:13:04.674 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-04T19:13:04.714 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-04T19:13:05.042 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-idna. 2024-09-04T19:13:05.068 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-04T19:13:05.108 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T19:13:05.352 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-urllib3. 2024-09-04T19:13:05.368 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-04T19:13:05.402 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T19:13:05.698 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests. 2024-09-04T19:13:05.724 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-04T19:13:05.764 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T19:13:06.100 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-04T19:13:06.127 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-04T19:13:06.166 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T19:13:06.428 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-04T19:13:06.443 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:06.486 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-modules-core (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:06.833 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-04T19:13:06.859 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:06.899 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:07.152 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-04T19:13:07.168 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-04T19:13:07.201 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-04T19:13:07.504 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-openssl. 2024-09-04T19:13:07.531 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-04T19:13:07.579 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-04T19:13:07.915 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr. 2024-09-04T19:13:07.942 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:07.982 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:08.343 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mon. 2024-09-04T19:13:08.370 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:08.409 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mon (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:09.756 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-osd. 2024-09-04T19:13:09.782 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:09.822 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-osd (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:10.858 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph. 2024-09-04T19:13:10.874 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../066-ceph_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:10.917 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:11.203 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-fuse. 2024-09-04T19:13:11.229 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:11.269 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:11.606 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mds. 2024-09-04T19:13:11.632 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:11.672 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mds (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:12.058 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephadm. 2024-09-04T19:13:12.085 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../069-cephadm_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:12.124 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:12.444 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-04T19:13:12.470 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-04T19:13:12.510 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-04T19:13:12.847 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-04T19:13:12.873 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T19:13:12.913 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T19:13:13.859 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-babel. 2024-09-04T19:13:13.875 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-04T19:13:13.909 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T19:13:14.154 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jinja2. 2024-09-04T19:13:14.180 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-04T19:13:14.212 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T19:13:14.515 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-04T19:13:14.542 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:14.598 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:14.876 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-04T19:13:14.892 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-04T19:13:14.934 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-04T19:13:15.245 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-routes. 2024-09-04T19:13:15.272 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-04T19:13:15.312 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T19:13:15.640 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-04T19:13:15.667 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:15.706 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-dashboard (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:16.953 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-04T19:13:16.969 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-04T19:13:17.011 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T19:13:17.540 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-joblib. 2024-09-04T19:13:17.567 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-04T19:13:17.606 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T19:13:17.926 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-04T19:13:17.953 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-04T19:13:17.992 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-04T19:13:18.295 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn. 2024-09-04T19:13:18.322 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-04T19:13:18.362 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T19:13:19.245 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-04T19:13:19.273 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:19.397 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:20.259 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cachetools. 2024-09-04T19:13:20.286 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-04T19:13:20.317 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-04T19:13:20.603 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-04T19:13:20.630 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-04T19:13:20.669 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-04T19:13:20.981 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-04T19:13:21.008 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-04T19:13:21.047 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T19:13:21.342 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rsa. 2024-09-04T19:13:21.369 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-04T19:13:21.400 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-04T19:13:21.687 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-google-auth. 2024-09-04T19:13:21.714 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-04T19:13:21.753 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-04T19:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-04T19:13:22.126 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-04T19:13:22.165 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T19:13:22.476 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-websocket. 2024-09-04T19:13:22.503 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-04T19:13:22.542 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-04T19:13:22.854 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-04T19:13:22.882 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-04T19:13:22.920 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T19:13:23.559 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-04T19:13:23.588 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:23.626 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:23.903 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-04T19:13:23.931 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-04T19:13:23.961 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T19:13:24.298 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-04T19:13:24.326 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T19:13:24.364 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T19:13:24.625 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package jq. 2024-09-04T19:13:24.641 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-04T19:13:24.683 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-04T19:13:24.927 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package socat. 2024-09-04T19:13:24.944 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-04T19:13:24.977 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-04T19:13:25.264 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package xmlstarlet. 2024-09-04T19:13:25.280 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-04T19:13:25.314 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-04T19:13:25.643 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-test. 2024-09-04T19:13:25.659 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../097-ceph-test_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:25.693 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-test (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:27.780 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-volume. 2024-09-04T19:13:27.796 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-235-g1ef86450-1jammy_all.deb ... 2024-09-04T19:13:27.838 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-volume (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:28.100 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-04T19:13:28.116 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:28.158 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs-dev (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:28.528 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-04T19:13:28.555 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-04T19:13:28.594 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T19:13:28.936 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-04T19:13:28.964 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-04T19:13:29.007 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-04T19:13:29.553 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package nvme-cli. 2024-09-04T19:13:29.580 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-04T19:13:29.610 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T19:13:29.958 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package pkg-config. 2024-09-04T19:13:29.985 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-04T19:13:30.024 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T19:13:30.302 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-04T19:13:30.329 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-04T19:13:30.368 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-04T19:13:30.780 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-04T19:13:30.807 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-04T19:13:30.846 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-04T19:13:31.140 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastescript. 2024-09-04T19:13:31.156 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-04T19:13:31.190 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-04T19:13:31.469 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pluggy. 2024-09-04T19:13:31.496 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-04T19:13:31.535 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-04T19:13:31.846 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-psutil. 2024-09-04T19:13:31.872 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-04T19:13:31.912 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-04T19:13:32.216 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-py. 2024-09-04T19:13:32.244 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-04T19:13:32.282 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-04T19:13:32.594 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pygments. 2024-09-04T19:13:32.622 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-04T19:13:32.660 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T19:13:33.148 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-04T19:13:33.164 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-04T19:13:33.205 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-04T19:13:33.508 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-toml. 2024-09-04T19:13:33.536 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-04T19:13:33.576 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-04T19:13:33.877 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pytest. 2024-09-04T19:13:33.905 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-04T19:13:33.943 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T19:13:34.289 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplejson. 2024-09-04T19:13:34.317 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-04T19:13:34.363 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-04T19:13:34.658 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-04T19:13:34.686 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-04T19:13:34.724 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-04T19:13:35.420 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package radosgw. 2024-09-04T19:13:35.448 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../116-radosgw_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:35.478 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking radosgw (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:36.264 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package rbd-fuse. 2024-09-04T19:13:36.280 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-235-g1ef86450-1jammy_amd64.deb ... 2024-09-04T19:13:36.322 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking rbd-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:36.602 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-04T19:13:36.869 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-04T19:13:36.995 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-04T19:13:37.262 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-04T19:13:37.602 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-04T19:13:38.000 INFO:teuthology.orchestra.run.smithi174.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-04T19:13:38.072 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:38.580 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user cephadm....done 2024-09-04T19:13:38.736 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-04T19:13:39.029 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-04T19:13:39.272 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-04T19:13:39.373 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-04T19:13:39.607 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-04T19:13:39.867 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-04T19:13:39.993 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-04T19:13:40.294 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-04T19:13:40.654 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-04T19:13:40.916 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-04T19:13:41.184 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-04T19:13:41.418 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-04T19:13:41.669 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-04T19:13:41.937 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-04T19:13:42.196 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-argparse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:42.447 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-04T19:13:42.623 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-04T19:13:42.749 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-04T19:13:42.875 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-04T19:13:43.134 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-04T19:13:43.537 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-04T19:13:43.565 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-04T19:13:43.629 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-04T19:13:43.746 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-04T19:13:44.098 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-04T19:13:44.224 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-04T19:13:44.508 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-04T19:13:44.776 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-04T19:13:45.469 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up zip (3.0-12build2) ... 2024-09-04T19:13:45.595 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-04T19:13:46.296 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-04T19:13:46.564 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-04T19:13:46.690 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-04T19:13:46.816 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-04T19:13:47.108 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-04T19:13:47.476 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-04T19:13:47.719 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-04T19:13:48.079 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-04T19:13:48.380 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-04T19:13:48.706 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-04T19:13:48.966 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-04T19:13:49.092 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:13:49.376 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-04T19:13:49.685 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-04T19:13:50.891 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-04T19:13:51.159 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:13:51.293 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-04T19:13:51.561 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-04T19:13:51.687 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-04T19:13:51.805 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-04T19:13:52.106 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-04T19:13:52.365 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-04T19:13:52.608 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:13:52.717 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-04T19:13:53.002 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-04T19:13:53.269 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-04T19:13:53.537 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-04T19:13:53.788 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-04T19:13:54.039 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-04T19:13:54.315 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-04T19:13:54.441 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-04T19:13:54.717 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-04T19:13:54.835 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-04T19:13:55.136 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-04T19:13:55.453 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-04T19:13:55.704 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-04T19:13:55.847 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-04T19:13:55.965 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-04T19:13:56.054 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-04T19:13:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-04T19:13:56.418 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-04T19:13:56.802 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-04T19:13:57.062 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-04T19:13:57.187 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-04T19:13:57.447 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-04T19:13:57.573 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-04T19:13:57.907 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-04T19:13:58.208 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-04T19:13:58.484 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-04T19:13:58.610 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-04T19:13:58.895 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-04T19:13:59.170 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-04T19:13:59.507 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-04T19:13:59.875 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-04T19:14:00.160 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-04T19:14:00.504 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-04T19:14:00.621 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librados2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:00.722 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libsqlite3-mod-ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:00.849 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-04T19:14:01.160 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:01.286 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libradosstriper1 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:01.412 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-04T19:14:01.688 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librbd1 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:01.814 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-modules-core (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:01.940 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:02.144 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T19:14:02.145 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-04T19:14:02.700 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs-dev (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:02.868 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rados (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:03.002 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librgw2 (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:03.128 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-04T19:14:04.374 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rbd (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:04.500 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up rbd-fuse (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:04.626 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rgw (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:04.752 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cephfs (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:04.878 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-common (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:05.412 INFO:teuthology.orchestra.run.smithi174.stdout:Adding group ceph....done 2024-09-04T19:14:05.780 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user ceph....done 2024-09-04T19:14:05.949 INFO:teuthology.orchestra.run.smithi174.stdout:Setting system user ceph properties....done 2024-09-04T19:14:05.965 INFO:teuthology.orchestra.run.smithi174.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-04T19:14:06.105 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-04T19:14:06.437 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-04T19:14:06.920 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-test (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:07.046 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up radosgw (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T19:14:07.524 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-04T19:14:07.956 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-base (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:08.306 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-04T19:14:08.775 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mds (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:08.994 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T19:14:08.994 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-04T19:14:09.459 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:09.675 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T19:14:09.675 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-04T19:14:10.144 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-osd (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:10.439 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T19:14:10.439 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-04T19:14:10.996 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-k8sevents (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:11.191 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:11.317 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mon (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:11.521 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T19:14:11.521 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-04T19:14:12.043 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-cephadm (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:12.194 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:12.320 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-dashboard (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:12.462 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-volume (19.1.1-235-g1ef86450-1jammy) ... 2024-09-04T19:14:12.832 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-04T19:14:12.949 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-04T19:14:13.093 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-04T19:14:14.127 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-04T19:14:14.786 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-04T19:14:14.792 DEBUG:teuthology.parallel:result is None 2024-09-04T19:14:14.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1ef864504b8875c83ee6c2c5fedc13315bebf7f5 2024-09-04T19:14:14.927 DEBUG:teuthology.orchestra.run.smithi174:> dpkg-query -W -f '${Version}' ceph 2024-09-04T19:14:14.946 INFO:teuthology.orchestra.run.smithi174.stdout:19.1.1-235-g1ef86450-1jammy 2024-09-04T19:14:14.946 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-235-g1ef86450-1jammy 2024-09-04T19:14:14.947 INFO:teuthology.task.install:The correct ceph version 19.1.1-235-g1ef86450-1jammy is installed. 2024-09-04T19:14:14.948 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-04T19:14:14.949 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:14:14.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-04T19:14:14.999 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-04T19:14:15.000 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:14:15.000 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-04T19:14:15.055 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-04T19:14:15.110 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-04T19:14:15.110 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:14:15.111 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-04T19:14:15.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-04T19:14:15.219 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-04T19:14:15.220 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:14:15.220 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-04T19:14:15.276 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-04T19:14:15.331 INFO:teuthology.run_tasks:Running task workunit... 2024-09-04T19:14:15.356 INFO:tasks.workunit:Pulling workunits from ref 055b029e365e266078e29644ed3ff217dfe73d04 2024-09-04T19:14:15.357 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-04T19:14:15.357 INFO:tasks.workunit:timeout=3h 2024-09-04T19:14:15.357 INFO:tasks.workunit:cleanup=True 2024-09-04T19:14:15.358 DEBUG:teuthology.orchestra.run.smithi174:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-04T19:14:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T19:14:15.380 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-04T19:14:15.380 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-04T19:14:15.428 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-04T19:14:15.428 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-04T19:14:15.476 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ljflores/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 055b029e365e266078e29644ed3ff217dfe73d04 2024-09-04T19:14:15.526 INFO:tasks.workunit.client.0.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-04T19:14:58.314 INFO:tasks.workunit.client.0.smithi174.stderr:Note: switching to '055b029e365e266078e29644ed3ff217dfe73d04'. 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:state without impacting any branches by switching back to a branch. 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: git switch -c 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr:Or undo this operation with: 2024-09-04T19:14:58.315 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.316 INFO:tasks.workunit.client.0.smithi174.stderr: git switch - 2024-09-04T19:14:58.316 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-04T19:14:58.316 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-04T19:14:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:HEAD is now at 055b029e365 qa/tasks/ceph_manager: set-require-min-compat-client to squid for msr profiles 2024-09-04T19:14:58.318 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-04T19:14:58.366 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-04T19:14:58.367 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-04T19:14:58.413 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-04T19:14:58.413 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-04T19:14:58.414 DEBUG:teuthology.orchestra.run.smithi174:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-04T19:14:58.478 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-04T19:14:58.509 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-04T19:14:58.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:14:58.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/bad-inc-map 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:47149 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:47149 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-04T19:14:58.510 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-04T19:14:58.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none ' 2024-09-04T19:14:58.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:47149 ' 2024-09-04T19:14:58.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-04T19:14:58.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-04T19:14:58.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-04T19:14:58.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:14:58.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:14:58.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:14:58.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:14:58.515 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:14:58.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:14:58.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:14:58.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:14:58.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:14:58.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:14:58.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:14:58.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:14:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:14:58.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:14:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:14:58.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:14:58.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-04T19:14:58.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:14:58.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:58.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:58.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21513 2024-09-04T19:14:58.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:14:58.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:14:58.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-09-04T19:14:58.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:14:58.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:58.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:58.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21513 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-04T19:14:58.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-09-04T19:14:58.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-04T19:14:59.031 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:14:59.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:14:59.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:14:59.032 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:14:59.033 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:59.033 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:59.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:14:59.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:14:59.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:14:59.213 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:14:59.214 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:14:59.214 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:14:59.214 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:59.215 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21513/ceph-mon.a.asok 2024-09-04T19:14:59.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:14:59.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21513/ceph-mon.a.asok config get fsid 2024-09-04T19:14:59.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:59.334 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21513/ceph-mon.a.asok 2024-09-04T19:14:59.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:14:59.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21513/ceph-mon.a.asok config get mon_host 2024-09-04T19:14:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-04T19:14:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-09-04T19:14:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:14:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:14:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:14:59.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-09-04T19:14:59.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:14:59.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:14:59.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:14:59.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:14:59.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:14:59.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:14:59.982 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:14:59.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:14:59.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:14:59.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:15:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-04T19:15:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-04T19:15:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:15:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:15:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 ' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:15:00.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:15:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:15:00.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-09-04T19:15:00.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:15:00.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78a9570f-aa51-45a1-bdaf-c9bda1850d21 2024-09-04T19:15:00.128 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 78a9570f-aa51-45a1-bdaf-c9bda1850d21 2024-09-04T19:15:00.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 78a9570f-aa51-45a1-bdaf-c9bda1850d21' 2024-09-04T19:15:00.129 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:15:00.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0sdhmKeqjCBAAqj1i/GtpXMzbP+cjB5J/+A== 2024-09-04T19:15:00.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0sdhmKeqjCBAAqj1i/GtpXMzbP+cjB5J/+A=="}' 2024-09-04T19:15:00.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78a9570f-aa51-45a1-bdaf-c9bda1850d21 -i td/bad-inc-map/0/new.json 2024-09-04T19:15:00.427 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:15:00.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-09-04T19:15:00.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0sdhmKeqjCBAAqj1i/GtpXMzbP+cjB5J/+A== --osd-uuid 78a9570f-aa51-45a1-bdaf-c9bda1850d21 2024-09-04T19:15:00.510 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:00.504+0000 7f04542517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:00.513 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:00.508+0000 7f04542517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:00.514 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:00.508+0000 7f04542517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:00.515 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:00.508+0000 7f04542517c0 -1 bdev(0x55709881ea80 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:15:00.515 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:00.508+0000 7f04542517c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-04T19:15:06.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-04T19:15:06.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:15:06.588 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:15:06.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:15:06.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:15:06.832 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:15:06.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:15:06.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:15:06.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:15:06.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:15:06.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:15:06.862 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:06.855+0000 7f6f72f337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:06.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:06.859+0000 7f6f72f337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:06.867 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:06.863+0000 7f6f72f337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:15:07.054 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:15:07.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:07.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:07.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:07.998 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:07.991+0000 7f6f72f337c0 -1 Falling back to public interface 2024-09-04T19:15:08.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:08.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:08.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:15:08.285 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:15:08.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:08.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:08.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:09.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:09.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:09.572 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:15:09.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:15:09.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:09.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:10.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:11.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:11.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:11.037 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:15:11.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:15:11.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:11.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:11.144 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:11.139+0000 7f6f72f337c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:15:11.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:12.573 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:15:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:15:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:12.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:13.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:14.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:14.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:14.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:15:14.018 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:15:14.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:14.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:14.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:15.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:15.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:15.436 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:15:15.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:15:15.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:15.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:15.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:16.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:16.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:16.891 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:15:16.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:15:16.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:16.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:17.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:18.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:18.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:18.363 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:15:18.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:15:18.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:18.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:18.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:19.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:19.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:19.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:15:19.820 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:15:19.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:19.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:20.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:21.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:21.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:21.262 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:15:21.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:15:21.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:21.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:21.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:22.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:22.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:22.670 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:15:22.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:15:22.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:22.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:23.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:24.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:24.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:24.183 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:15:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:15:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:24.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:25.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:25.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:25.843 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:15:25.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:15:25.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:25.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:26.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:27.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:27.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:27.875 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:15:27.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:15:27.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:27.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:29.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:30.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:30.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:30.079 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:15:30.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:15:30.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:30.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:30.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:31.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:31.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:31.701 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:15:31.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:15:31.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:31.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:32.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:33.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:33.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:33.683 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:15:33.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:15:33.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:33.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:34.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:35.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:35.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:35.216 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:15:35.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:15:35.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:35.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:15:35.725 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/527478806,v1:127.0.0.1:6801/527478806] [v2:127.0.0.1:6802/527478806,v1:127.0.0.1:6803/527478806] exists,up 78a9570f-aa51-45a1-bdaf-c9bda1850d21 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 ' 2024-09-04T19:15:35.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:15:35.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-04T19:15:35.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:15:35.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-09-04T19:15:35.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:15:35.762 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 61e732c4-7d24-4797-93d7-a03582eeedcb 2024-09-04T19:15:35.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61e732c4-7d24-4797-93d7-a03582eeedcb 2024-09-04T19:15:35.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 61e732c4-7d24-4797-93d7-a03582eeedcb' 2024-09-04T19:15:35.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:15:35.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXsdhmRJGILhAA6UsNZbQ0fmK/sJdebQXktA== 2024-09-04T19:15:35.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXsdhmRJGILhAA6UsNZbQ0fmK/sJdebQXktA=="}' 2024-09-04T19:15:35.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61e732c4-7d24-4797-93d7-a03582eeedcb -i td/bad-inc-map/1/new.json 2024-09-04T19:15:36.293 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:15:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-09-04T19:15:36.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXsdhmRJGILhAA6UsNZbQ0fmK/sJdebQXktA== --osd-uuid 61e732c4-7d24-4797-93d7-a03582eeedcb 2024-09-04T19:15:36.342 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:36.335+0000 7f6c0bfa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:36.363 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:36.359+0000 7f6c0bfa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:36.364 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:36.359+0000 7f6c0bfa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:36.365 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:36.359+0000 7f6c0bfa37c0 -1 bdev(0x55d8e38f8a80 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:15:36.365 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:36.359+0000 7f6c0bfa37c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-04T19:15:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-04T19:15:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:15:42.268 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:15:42.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:15:42.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:15:42.831 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:15:42.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:15:42.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:15:42.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:15:42.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:15:42.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:15:42.860 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:42.855+0000 7ff9e6ba77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:42.883 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:42.879+0000 7ff9e6ba77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:42.885 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:42.879+0000 7ff9e6ba77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:15:43.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:15:43.235 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:15:43.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:43.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:43.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:44.142 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:44.135+0000 7ff9e6ba77c0 -1 Falling back to public interface 2024-09-04T19:15:44.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:44.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:44.641 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:15:44.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:15:44.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:44.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:45.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:46.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:46.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:46.044 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:15:46.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:15:46.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:46.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:46.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:47.001 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:15:46.995+0000 7ff9e6ba77c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:15:47.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:47.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:47.452 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:15:47.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:15:47.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:47.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:48.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:49.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:49.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:49.156 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:15:49.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:15:49.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:49.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:49.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:50.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:50.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:50.610 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:15:50.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:15:50.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:50.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:51.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:52.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:52.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:52.193 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:15:52.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:15:52.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:52.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:52.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:53.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:53.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:53.624 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:15:53.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:15:53.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:53.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:54.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:55.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:55.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:55.124 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:15:55.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:15:55.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:55.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:55.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:56.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:56.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:56.558 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:15:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:15:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:56.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:57.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:57.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:57.981 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:15:57.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:15:57.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:57.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:15:58.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:15:59.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:15:59.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:15:59.383 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:15:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:15:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:15:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:00.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:01.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:01.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:01.536 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:16:01.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:16:01.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:01.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:02.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:03.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:03.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:03.315 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:16:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:16:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:04.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:05.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:05.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:05.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:16:05.395 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:16:05.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:05.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:06.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:07.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:07.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:07.060 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:16:07.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:16:07.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:07.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:07.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:08.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:08.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:08.549 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:16:08.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:16:08.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:08.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:10.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:10.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:10.781 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:16:10.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:16:10.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:10.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:11.198 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3147830672,v1:127.0.0.1:6811/3147830672] [v2:127.0.0.1:6812/3147830672,v1:127.0.0.1:6813/3147830672] exists,up 61e732c4-7d24-4797-93d7-a03582eeedcb 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-04T19:16:11.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 ' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:16:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:16:11.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:16:11.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-09-04T19:16:11.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:16:11.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=312e4036-b523-4728-9a77-22f69c95e9c8 2024-09-04T19:16:11.206 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 312e4036-b523-4728-9a77-22f69c95e9c8 2024-09-04T19:16:11.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 312e4036-b523-4728-9a77-22f69c95e9c8' 2024-09-04T19:16:11.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:16:11.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7sdhm3+5eDRAAxmjZ9RK9+2v8jLK+HrXmyQ== 2024-09-04T19:16:11.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7sdhm3+5eDRAAxmjZ9RK9+2v8jLK+HrXmyQ=="}' 2024-09-04T19:16:11.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 312e4036-b523-4728-9a77-22f69c95e9c8 -i td/bad-inc-map/2/new.json 2024-09-04T19:16:11.756 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:16:11.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-09-04T19:16:11.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7sdhm3+5eDRAAxmjZ9RK9+2v8jLK+HrXmyQ== --osd-uuid 312e4036-b523-4728-9a77-22f69c95e9c8 2024-09-04T19:16:11.798 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:11.794+0000 7fa80c21d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:11.807 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:11.802+0000 7fa80c21d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:11.808 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:11.806+0000 7fa80c21d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:11.809 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:11.806+0000 7fa80c21d7c0 -1 bdev(0x55e993974a80 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:16:11.809 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:11.806+0000 7fa80c21d7c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-04T19:16:18.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-04T19:16:18.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:16:18.004 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:16:18.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:16:18.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:16:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:16:18.556 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:16:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=932cdc43-017b-4b90-bb65-3cfa1186a817 --auth-supported=none --mon-host=127.0.0.1:47149 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21513/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:16:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:16:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:16:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:16:18.585 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:18.582+0000 7fe3446de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:18.612 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:18.610+0000 7fe3446de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:18.613 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:18.610+0000 7fe3446de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:19.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:16:19.051 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:16:19.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:19.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:19.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:19.714 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:19.710+0000 7fe3446de7c0 -1 Falling back to public interface 2024-09-04T19:16:20.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:20.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:20.510 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:16:20.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:16:20.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:20.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:20.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:21.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:21.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:21.952 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:16:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:16:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:22.274 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:16:22.270+0000 7fe3446de7c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:16:22.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:23.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:23.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:23.366 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:16:23.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:16:23.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:23.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:23.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:24.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:24.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:24.829 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:16:24.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:16:24.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:24.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:25.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:26.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:26.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:26.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:16:26.371 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:16:26.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:26.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:26.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:27.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:27.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:27.793 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:16:27.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:16:27.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:27.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:28.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:29.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:29.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:29.320 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:16:29.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:16:29.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:29.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:29.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:30.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:30.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:30.764 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:16:30.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:16:30.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:30.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:31.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:32.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:32.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:32.282 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:16:32.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:16:32.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:32.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:32.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:33.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:33.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:33.801 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:16:33.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:16:33.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:33.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:34.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:35.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:35.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:35.264 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:16:35.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:16:35.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:35.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:35.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:36.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:36.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:36.798 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:16:36.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:16:36.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:36.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:37.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:38.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:38.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:38.660 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:16:38.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:16:38.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:38.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:39.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:40.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:40.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:40.997 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:16:40.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:16:40.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:40.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:41.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:42.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:42.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:42.945 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:16:42.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:16:42.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:42.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:43.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:16:44.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:16:44.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:16:44.912 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:16:44.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:16:44.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:44.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:45.331 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/628559311,v1:127.0.0.1:6819/628559311] [v2:127.0.0.1:6820/628559311,v1:127.0.0.1:6821/628559311] exists,up 312e4036-b523-4728-9a77-22f69c95e9c8 2024-09-04T19:16:45.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:16:45.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:16:45.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:16:45.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-04T19:16:45.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-04T19:16:45.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-09-04T19:16:46.397 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'foo' already exists 2024-09-04T19:16:46.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:16:47.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-04T19:16:48.876 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 min_size to 1 2024-09-04T19:16:48.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-04T19:16:49.992 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 min_size to 2 2024-09-04T19:16:50.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:16:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-04T19:16:55.025 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:16:55.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:16:55.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:55.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:16:55.480 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/527478806,v1:127.0.0.1:6801/527478806] [v2:127.0.0.1:6802/527478806,v1:127.0.0.1:6803/527478806] exists,up 78a9570f-aa51-45a1-bdaf-c9bda1850d21 2024-09-04T19:16:55.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:16:55.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:16:55.482 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:16:55.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:55.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:16:55.939 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3147830672,v1:127.0.0.1:6811/3147830672] [v2:127.0.0.1:6812/3147830672,v1:127.0.0.1:6813/3147830672] exists,up 61e732c4-7d24-4797-93d7-a03582eeedcb 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:16:55.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:16:55.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-04T19:16:55.941 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:16:55.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:16:55.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:16:55.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:16:56.382 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/628559311,v1:127.0.0.1:6819/628559311] [v2:127.0.0.1:6820/628559311,v1:127.0.0.1:6821/628559311] exists,up 312e4036-b523-4728-9a77-22f69c95e9c8 2024-09-04T19:16:56.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:16:56.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:16:56.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:16:56.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.246+0000 7fe335682640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.534+0000 7fe335682640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.910+0000 7fe335682640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:47.706+0000 7fe33367e640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:47.926+0000 7fe335682640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.274+0000 7fe33367e640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.274+0000 7fe33367e640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.590+0000 7fe33367e640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.042+0000 7fe33367e640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.146+0000 7fe33367e640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.278+0000 7fe335682640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.386 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.278+0000 7fe335682640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.387 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.702+0000 7fe335682640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.387 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:50.050+0000 7fe335682640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.387 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:50.446+0000 7fe335682640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-09-04T19:16:56.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.250+0000 7fe335682640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.534+0000 7fe335682640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:46.910+0000 7fe335682640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:47.706+0000 7fe33367e640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:47.926+0000 7fe335682640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-09-04T19:16:56.388 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.274+0000 7fe33367e640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.274+0000 7fe33367e640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:48.590+0000 7fe33367e640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.042+0000 7fe33367e640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.146+0000 7fe33367e640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.278+0000 7fe335682640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.282+0000 7fe335682640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:49.702+0000 7fe335682640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:50.050+0000 7fe335682640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:16:50.446+0000 7fe335682640 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2024-09-04T19:16:56.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-04T19:16:56.390 INFO:tasks.workunit.client.0.smithi174.stdout:success 2024-09-04T19:16:56.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-04T19:16:56.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-09-04T19:16:56.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-04T19:16:56.961 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'foo' does not exist 2024-09-04T19:16:56.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-04T19:16:56.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:16:56.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:16:56.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:16:56.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:16:56.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:17:02.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:17:02.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-04T19:17:02.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:17:02.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:17:02.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:17:02.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:17:02.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:17:02.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:17:02.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:17:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:17:02.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:17:02.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:17:02.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:17:02.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:17:02.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:17:02.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-04T19:17:02.326 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:17:02.326 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:02.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:17:02.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21513 2024-09-04T19:17:02.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:17:02.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:17:02.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-04T19:17:02.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/bad-inc-map 0 2024-09-04T19:17:02.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:17:02.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:17:02.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:17:02.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:17:02.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:17:02.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:17:02.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:17:02.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:17:02.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:17:02.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:17:02.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:17:02.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:17:02.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-04T19:17:02.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-04T19:17:02.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21513 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21513 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:17:02.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-04T19:17:02.339 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-04T19:17:02.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-04T19:17:02.389 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-04T19:17:02.389 DEBUG:teuthology.orchestra.run.smithi174:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-04T19:17:02.442 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/divergent-priors 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-04T19:17:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-04T19:17:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-04T19:17:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-04T19:17:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-04T19:17:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-04T19:17:02.452 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-04T19:17:02.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-04T19:17:02.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-04T19:17:02.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none ' 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-04T19:17:02.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_divergent_2 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_divergent_3 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_divergent_ec' 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-04T19:17:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:17:02.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:17:02.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:17:02.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:17:02.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:17:02.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:17:02.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:17:02.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:17:02.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:17:02.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:17:02.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:17:02.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:17:02.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:17:02.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:17:02.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:17:02.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:17:02.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:02.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:02.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:17:02.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:17:02.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:17:02.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-04T19:17:02.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:17:02.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:02.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:02.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27364 2024-09-04T19:17:02.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:17:02.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-04T19:17:02.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-04T19:17:02.473 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-04T19:17:02.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-04T19:17:02.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-04T19:17:02.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:02.863 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:02.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:02.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:17:03.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:17:03.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:17:03.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:17:03.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:17:03.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:17:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:03.075 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:03.075 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:17:03.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:17:03.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get fsid 2024-09-04T19:17:03.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:17:03.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:17:03.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:17:03.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:17:03.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:17:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:17:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:17:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:17:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:17:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:17:03.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:17:03.193 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get mon_host 2024-09-04T19:17:03.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-04T19:17:03.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:17:04.211 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:04.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:17:04.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:17:04.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-04T19:17:04.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-04T19:17:04.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:17:04.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:17:04.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:17:04.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:17:04.260 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:17:04.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79f12bb7-8ec8-4178-aacb-b507f8d310dc' 2024-09-04T19:17:04.260 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:17:04.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwsthmbsayEBAAcmNVXLr9Tat8hysYB6Rzxw== 2024-09-04T19:17:04.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwsthmbsayEBAAcmNVXLr9Tat8hysYB6Rzxw=="}' 2024-09-04T19:17:04.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79f12bb7-8ec8-4178-aacb-b507f8d310dc -i td/divergent-priors/0/new.json 2024-09-04T19:17:04.672 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:17:04.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-04T19:17:04.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwsthmbsayEBAAcmNVXLr9Tat8hysYB6Rzxw== --osd-uuid 79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:17:04.731 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:04.726+0000 7fa8369dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:04.754 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:04.750+0000 7fa8369dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:04.755 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:04.750+0000 7fa8369dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:04.756 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:04.754+0000 7fa8369dc7c0 -1 bdev(0x562628814a80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:17:04.756 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:04.754+0000 7fa8369dc7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-04T19:17:10.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-04T19:17:10.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:17:10.686 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:17:10.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:17:10.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:17:11.317 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:17:11.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:17:11.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:17:11.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:17:11.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:17:11.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:17:11.358 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:11.353+0000 7fa1500e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:11.357+0000 7fa1500e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:11.362 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:11.357+0000 7fa1500e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:11.731 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:17:11.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:17:11.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:11.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:12.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:12.178 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:12.173+0000 7fa1500e77c0 -1 Falling back to public interface 2024-09-04T19:17:13.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:13.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:13.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:17:13.155 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:17:13.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:13.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:13.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:14.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:14.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:14.579 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:17:14.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:17:14.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:14.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:15.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:15.218 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:15.213+0000 7fa1500e77c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:17:16.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:16.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:16.020 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:17:16.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:17:16.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:16.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:16.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:17.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:17.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:17.592 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:17:17.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:17:17.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:17.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:18.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:19.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:19.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:19.074 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:17:19.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:17:19.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:19.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:19.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:20.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:20.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:20.533 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:17:20.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:17:20.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:20.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:21.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:22.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:22.076 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:17:22.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:22.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:17:22.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:22.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:22.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:23.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:23.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:23.515 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:17:23.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:17:23.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:23.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:24.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:25.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:25.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:25.039 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:17:25.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:17:25.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:25.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:25.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:26.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:26.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:26.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:17:26.492 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:17:26.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:26.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:27.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:28.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:28.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:28.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:17:28.034 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:17:28.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:28.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:28.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:29.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:29.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:29.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:17:29.574 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:17:29.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:29.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:30.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:31.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:31.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:31.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:17:31.218 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:17:31.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:31.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:32.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:33.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:33.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:33.194 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:17:33.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:17:33.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:33.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:34.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:35.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:35.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:35.055 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:17:35.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:17:35.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:35.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:36.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:37.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:37.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:37.364 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:17:37.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:17:37.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:37.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:17:37.787 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2322169630,v1:127.0.0.1:6803/2322169630] [v2:127.0.0.1:6804/2322169630,v1:127.0.0.1:6805/2322169630] exists,up 79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:17:37.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:17:37.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:17:37.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:17:37.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:17:37.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:17:37.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:17:37.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:17:37.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:17:37.795 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:17:37.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2c064751-aed6-4a4d-a8d9-0a69663e2801' 2024-09-04T19:17:37.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:17:37.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRsthmF5PaMBAAiv1/IJfJEMOAQphQ+glE6g== 2024-09-04T19:17:37.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRsthmF5PaMBAAiv1/IJfJEMOAQphQ+glE6g=="}' 2024-09-04T19:17:37.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c064751-aed6-4a4d-a8d9-0a69663e2801 -i td/divergent-priors/1/new.json 2024-09-04T19:17:38.317 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:17:38.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-04T19:17:38.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRsthmF5PaMBAAiv1/IJfJEMOAQphQ+glE6g== --osd-uuid 2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:17:38.376 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:38.373+0000 7efc7fd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:38.377 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:38.373+0000 7efc7fd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:38.379 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:38.373+0000 7efc7fd287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:38.379 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:38.373+0000 7efc7fd287c0 -1 bdev(0x556896e4ca80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:17:38.379 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:38.373+0000 7efc7fd287c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-04T19:17:44.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-04T19:17:44.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:17:44.344 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:17:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:17:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:17:44.905 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:17:44.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:17:44.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:17:44.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:17:44.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:17:44.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:17:44.945 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:44.941+0000 7fd986ac87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:44.948 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:44.945+0000 7fd986ac87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:44.949 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:44.945+0000 7fd986ac87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:17:45.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:45.388 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:17:45.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:17:45.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:45.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:45.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:45.998 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:45.993+0000 7fd986ac87c0 -1 Falling back to public interface 2024-09-04T19:17:46.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:46.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:46.801 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:17:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:17:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:47.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:48.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:48.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:48.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:17:48.240 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:17:48.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:48.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:48.596 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:17:48.593+0000 7fd986ac87c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:17:48.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:49.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:49.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:49.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:17:49.664 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:17:49.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:49.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:50.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:51.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:51.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:51.122 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:17:51.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:17:51.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:51.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:51.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:52.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:52.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:52.763 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:17:52.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:17:52.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:52.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:53.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:54.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:54.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:54.357 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:17:54.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:17:54.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:54.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:54.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:55.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:55.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:55.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:17:55.797 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:17:55.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:55.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:56.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:57.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:57.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:57.416 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:17:57.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:17:57.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:57.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:57.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:17:58.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:17:58.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:17:58.993 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:17:58.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:17:58.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:17:58.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:17:59.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:00.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:18:00.482 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:18:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:00.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:01.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:18:01.920 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:18:01.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:01.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:02.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:03.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:03.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:03.990 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:18:03.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:18:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:05.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:06.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:06.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:06.091 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:18:06.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:18:06.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:06.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:07.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:08.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:08.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:08.529 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:18:08.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:18:08.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:08.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:10.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:10.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:10.331 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:18:10.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:18:10.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:10.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:11.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:12.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:12.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:12.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:18:12.089 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:18:12.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:12.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:18:12.569 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2805168138,v1:127.0.0.1:6811/2805168138] [v2:127.0.0.1:6812/2805168138,v1:127.0.0.1:6813/2805168138] exists,up 2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:18:12.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:18:12.573 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:18:12.574 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:18:12.575 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:18:12.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:18:12.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:18:12.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:18:12.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:18:12.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:18:12.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:18:12.577 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:18:12.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f23fe3fd-faa7-40fe-a7bd-945126d4d79f' 2024-09-04T19:18:12.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:18:12.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0sthm+/DNIxAAWxDaF7LiC8QTWUcOr3Tv0A== 2024-09-04T19:18:12.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0sthm+/DNIxAAWxDaF7LiC8QTWUcOr3Tv0A=="}' 2024-09-04T19:18:12.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f23fe3fd-faa7-40fe-a7bd-945126d4d79f -i td/divergent-priors/2/new.json 2024-09-04T19:18:13.099 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:18:13.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-04T19:18:13.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0sthm+/DNIxAAWxDaF7LiC8QTWUcOr3Tv0A== --osd-uuid f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:18:13.161 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:13.156+0000 7f9ff5a7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:13.163 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:13.156+0000 7f9ff5a7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:13.164 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:13.160+0000 7f9ff5a7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:13.165 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:13.160+0000 7f9ff5a7e7c0 -1 bdev(0x55c226d3aa80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:18:13.165 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:13.160+0000 7f9ff5a7e7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-04T19:18:19.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-04T19:18:19.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:18:19.347 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:18:19.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:18:19.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:18:19.939 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:18:19.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:18:19.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:18:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:18:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:18:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:18:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:19.972+0000 7f00b76e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:19.988 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:19.984+0000 7f00b76e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:19.989 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:19.984+0000 7f00b76e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:18:20.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:18:20.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:18:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:20.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:20.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:21.238 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:21.232+0000 7f00b76e07c0 -1 Falling back to public interface 2024-09-04T19:18:21.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:21.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:21.866 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:18:21.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:18:21.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:21.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:22.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:23.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:23.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:23.284 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:18:23.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:18:23.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:23.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:23.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:24.030 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:18:24.024+0000 7f00b76e07c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:18:24.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:24.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:24.728 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:18:24.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:18:24.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:24.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:25.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:26.342 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:18:26.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:18:26.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:26.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:26.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:27.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:27.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:27.822 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:18:27.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:18:27.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:27.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:29.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:29.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:29.241 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:18:29.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:18:29.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:29.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:29.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:30.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:30.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:30.799 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:18:30.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:18:30.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:30.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:31.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:32.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:32.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:32.220 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:18:32.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:18:32.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:32.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:32.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:33.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:33.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:33.652 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:18:33.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:18:33.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:33.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:34.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:35.235 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:18:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:18:35.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:35.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:35.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:36.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:36.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:36.845 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:18:36.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:18:36.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:36.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:37.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:38.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:38.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:38.313 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:18:38.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:18:38.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:38.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:38.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:39.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:39.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:39.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:18:39.927 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:18:39.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:39.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:40.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:41.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:41.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:41.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:18:41.590 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:18:41.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:41.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:43.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:43.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:43.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:18:43.528 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:18:43.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:43.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:44.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:45.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:45.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:45.457 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:18:45.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:18:45.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:45.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:45.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:18:46.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:18:46.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:18:46.936 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:18:46.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:18:46.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:18:46.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:18:47.360 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1532273204,v1:127.0.0.1:6819/1532273204] [v2:127.0.0.1:6820/1532273204,v1:127.0.0.1:6821/1532273204] exists,up f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:18:47.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:18:47.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:18:47.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:18:47.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-04T19:18:48.028 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-04T19:18:48.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-04T19:18:48.807 INFO:tasks.workunit.client.0.smithi174.stderr:noin is set 2024-09-04T19:18:48.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-04T19:18:49.560 INFO:tasks.workunit.client.0.smithi174.stderr:nodown is set 2024-09-04T19:18:49.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-04T19:18:49.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-04T19:18:50.239 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' already exists 2024-09-04T19:18:50.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:18:51.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-04T19:18:51.997 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 size to 3 2024-09-04T19:18:52.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-04T19:18:52.682 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 min_size to 2 2024-09-04T19:18:52.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-04T19:18:52.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:18:52.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:18:53.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:18:53.140 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:18:53.140 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:18:53.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:18:53.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:53.141 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:18:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-09-04T19:18:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-09-04T19:18:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-09-04T19:18:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:53.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:18:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-09-04T19:18:53.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-09-04T19:18:53.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672972' 2024-09-04T19:18:53.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:53.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:18:53.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-09-04T19:18:53.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-09-04T19:18:53.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672972 2-64424509446' 2024-09-04T19:18:53.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:53.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-09-04T19:18:53.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:53.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:18:53.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-09-04T19:18:53.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:53.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-09-04T19:18:53.627 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836502 2024-09-04T19:18:53.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-04T19:18:53.627 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:18:54.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-04T19:18:54.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:18:55.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:18:55.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:18:55.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-04T19:18:55.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:55.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-09-04T19:18:55.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:55.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:18:55.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-09-04T19:18:55.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:55.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-09-04T19:18:55.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-04T19:18:55.466 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672972 2024-09-04T19:18:55.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:18:55.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-04T19:18:55.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:55.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-09-04T19:18:55.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:55.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:18:55.929 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-09-04T19:18:55.929 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:55.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-09-04T19:18:55.931 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509446 2024-09-04T19:18:55.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-04T19:18:55.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:18:56.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-04T19:18:56.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-04T19:18:56.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:18:56.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:18:56.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:18:56.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:18:56.359 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:18:56.359 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:18:56.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:18:56.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:18:56.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:18:56.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:18:56.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:18:56.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:18:56.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:18:56.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:18:56.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:18:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:18:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:18:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:18:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:18:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:56.876 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:18:57.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-09-04T19:18:57.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-09-04T19:18:57.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-09-04T19:18:57.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:57.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:18:57.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-09-04T19:18:57.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-09-04T19:18:57.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-09-04T19:18:57.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:18:57.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:18:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-09-04T19:18:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-09-04T19:18:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509449' 2024-09-04T19:18:57.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:57.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-09-04T19:18:57.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:57.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:18:57.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-09-04T19:18:57.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:57.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-09-04T19:18:57.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-04T19:18:57.359 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836504 2024-09-04T19:18:57.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:18:57.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-04T19:18:57.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:18:58.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:18:58.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:18:59.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-04T19:18:59.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:59.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-09-04T19:18:59.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:59.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:18:59.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-09-04T19:18:59.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:59.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-09-04T19:18:59.253 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672975 2024-09-04T19:18:59.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-04T19:18:59.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:18:59.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-04T19:18:59.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:18:59.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:18:59.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-09-04T19:18:59.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:18:59.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-09-04T19:18:59.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:18:59.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-09-04T19:18:59.732 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509449 2024-09-04T19:18:59.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-04T19:18:59.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:19:00.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-04T19:19:00.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:19:00.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:00.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:00.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:19:00.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:00.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:01.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:19:01.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:01.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:01.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:01.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:19:01.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:19:01.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:19:01.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-04T19:19:01.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-04T19:19:02.032 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:19:02.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-04T19:19:02.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-04T19:19:02.050 INFO:tasks.workunit.client.0.smithi174.stdout:primary and soon to be divergent is 1 2024-09-04T19:19:02.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-04T19:19:02.432 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:19:02.432 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-04T19:18:51.645146+0000 0'0 28:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T03:09:59.680724+0000 0 0 2024-09-04T19:19:02.433 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:19:02.433 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:19:02.433 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-04T19:19:02.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-04T19:19:02.449 INFO:tasks.workunit.client.0.smithi174.stdout:writing initial objects 2024-09-04T19:19:02.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-04T19:19:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:02.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-04T19:19:02.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:02.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-04T19:19:02.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:02.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-04T19:19:02.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:02.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-04T19:19:03.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-04T19:19:03.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-04T19:19:03.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-04T19:19:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-04T19:19:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-04T19:19:03.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-04T19:19:03.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:03.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-04T19:19:04.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-04T19:19:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-04T19:19:04.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-04T19:19:04.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-04T19:19:04.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-04T19:19:04.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-04T19:19:04.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:04.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-04T19:19:05.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:05.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-04T19:19:05.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:05.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-04T19:19:05.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:05.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-04T19:19:05.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:05.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-04T19:19:05.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:05.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-04T19:19:06.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-04T19:19:06.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-04T19:19:06.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-04T19:19:06.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-04T19:19:06.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-04T19:19:06.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-04T19:19:06.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:06.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-04T19:19:07.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-04T19:19:07.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-04T19:19:07.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-04T19:19:07.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-04T19:19:07.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-04T19:19:07.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-04T19:19:07.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:07.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-04T19:19:08.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-04T19:19:08.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-04T19:19:08.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-04T19:19:08.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-04T19:19:08.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-04T19:19:08.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:08.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-04T19:19:09.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-04T19:19:09.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-04T19:19:09.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-04T19:19:09.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-04T19:19:09.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-04T19:19:09.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-04T19:19:09.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:09.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-04T19:19:10.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-04T19:19:10.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-04T19:19:10.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-04T19:19:10.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-04T19:19:10.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-04T19:19:10.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:10.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-04T19:19:11.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-04T19:19:11.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-04T19:19:11.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-04T19:19:11.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-04T19:19:11.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-04T19:19:11.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-04T19:19:11.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:11.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-04T19:19:12.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-04T19:19:12.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-04T19:19:12.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-04T19:19:12.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-04T19:19:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-04T19:19:12.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:12.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-04T19:19:13.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-04T19:19:13.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-04T19:19:13.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-04T19:19:13.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-04T19:19:13.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-04T19:19:13.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-04T19:19:13.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:13.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-04T19:19:14.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-04T19:19:14.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-04T19:19:14.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-04T19:19:14.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-04T19:19:14.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-04T19:19:14.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-04T19:19:14.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:14.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-04T19:19:15.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-04T19:19:15.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-04T19:19:15.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-04T19:19:15.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-04T19:19:15.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-04T19:19:15.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:15.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-04T19:19:16.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-04T19:19:16.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-04T19:19:16.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-04T19:19:16.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-04T19:19:16.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-04T19:19:16.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:16.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-04T19:19:17.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:17.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-04T19:19:17.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:17.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-04T19:19:17.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:17.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-04T19:19:17.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:17.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-04T19:19:17.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-04T19:19:17.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-04T19:19:17.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:19:17.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-04T19:19:17.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:19:17.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:19:17.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:19:17.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:19:17.774 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:19:17.774 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:19:17.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:19:17.775 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:19:17.775 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:19:17.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:19:17.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:19:17.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:19:17.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:19:17.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:19:17.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:19:18.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:19:18.272 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:19:18.272 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:19:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:19:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:18.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:19:18.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-09-04T19:19:18.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-09-04T19:19:18.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-09-04T19:19:18.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:18.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:19:18.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-09-04T19:19:18.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-09-04T19:19:18.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672981' 2024-09-04T19:19:18.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:18.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:19:18.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-09-04T19:19:18.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-09-04T19:19:18.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672981 2-64424509455' 2024-09-04T19:19:18.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:19:18.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-09-04T19:19:18.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:19:18.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:19:18.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-09-04T19:19:18.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:19:18.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-09-04T19:19:18.766 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836511 2024-09-04T19:19:18.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-04T19:19:18.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:19:19.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-04T19:19:19.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:19:19.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-09-04T19:19:19.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:19:19.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:19:19.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-09-04T19:19:19.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:19:19.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-09-04T19:19:19.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-04T19:19:19.179 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672981 2024-09-04T19:19:19.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:19:19.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-04T19:19:19.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:19:19.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:19:19.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-09-04T19:19:19.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:19:19.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-09-04T19:19:19.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:19:19.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-09-04T19:19:19.622 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509455 2024-09-04T19:19:19.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-04T19:19:19.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:19:20.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-04T19:19:20.042 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:19:20.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:20.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:20.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:19:20.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:20.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:20.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:20.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:20.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:20.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:20.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:20.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:19:20.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:20.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:20.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:21.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:19:21.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:19:21.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:19:21.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-04T19:19:21.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-04T19:19:21.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-04T19:19:21.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-04T19:19:21.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-04T19:19:21.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-04T19:19:21.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-04T19:19:21.963 INFO:tasks.workunit.client.0.smithi174.stdout:blackholing osds 0 2 2024-09-04T19:19:21.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-04T19:19:22.338 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:19:22.338 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-04T19:18:51.645146+0000 28'100 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T03:09:59.680724+0000 0 0 2024-09-04T19:19:22.338 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:19:22.338 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:19:22.339 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:19:22.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:22.355 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:22.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:19:22.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-04T19:19:22.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-04T19:19:22.451 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:19:22.451 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:19:22.451 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:19:22.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:22.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-04T19:19:22.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:19:22.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-04T19:19:22.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-04T19:19:22.568 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:19:22.568 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:19:22.569 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:19:22.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-04T19:19:22.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-04T19:19:22.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-04T19:19:22.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-04T19:19:22.583 INFO:tasks.workunit.client.0.smithi174.stdout:writing divergent object 2024-09-04T19:19:22.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-04T19:19:22.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-04T19:19:22.583 INFO:tasks.workunit.client.0.smithi174.stdout:create missing divergent object 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=test 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=existing_99 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/divergent-priors 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-04T19:19:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds test existing_99 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-04T19:19:22.586 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-09-04T19:19:22.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-04T19:19:22.989 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-04T19:19:22.989 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0' '2') 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-09-04T19:19:22.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/divergent-priors/1/type 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-04T19:19:22.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-04T19:19:22.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:19:22.993 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27364/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-04T19:19:23.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-04T19:19:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-09-04T19:19:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-09-04T19:19:23.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-09-04T19:19:23.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:19:23.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-09-04T19:19:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok injectdataerr test existing_99 2024-09-04T19:19:23.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-04T19:19:23.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-04T19:19:33.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-04T19:19:33.234 INFO:tasks.workunit.client.0.smithi174.stderr:rados: no process found 2024-09-04T19:19:33.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-04T19:19:33.235 INFO:tasks.workunit.client.0.smithi174.stdout:killing all the osds 2024-09-04T19:19:33.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-04T19:19:33.645 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:19:33.645 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-09-04T19:19:23.421153+0000 28'101 28:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T03:09:59.680724+0000 0 0 2024-09-04T19:19:33.645 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:19:33.645 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:19:33.645 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:19:33.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-04T19:19:33.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:19:33.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:19:33.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:19:33.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:19:33.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:19:33.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:19:33.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-04T19:19:33.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-04T19:19:34.305 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:19:34.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-04T19:19:34.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-04T19:19:34.912 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:19:34.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-04T19:19:34.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-04T19:19:35.466 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:19:35.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:35.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-04T19:19:36.026 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-04T19:19:36.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:36.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-04T19:19:36.644 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-04T19:19:36.663 INFO:tasks.workunit.client.0.smithi174.stdout:bringing up non_divergent 0 2 2024-09-04T19:19:36.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-04T19:19:36.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-04T19:19:37.064 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:19:37.064 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-09-04T19:19:23.421153+0000 28'101 28:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T03:09:59.680724+0000 0 0 2024-09-04T19:19:37.064 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:19:37.064 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:19:37.064 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:19:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-04T19:19:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:19:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:19:37.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:37.083 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:37.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:19:37.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:19:37.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:19:37.086 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:19:37.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:19:37.123 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:37.116+0000 7f1d2538c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:37.125 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:37.120+0000 7f1d2538c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:37.126 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:37.120+0000 7f1d2538c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:37.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:19:37.487 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:19:37.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:37.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:19:37.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:38.426 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:38.420+0000 7f1d2538c7c0 -1 Falling back to public interface 2024-09-04T19:19:38.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:38.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:38.894 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:19:38.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:19:38.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:38.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:19:39.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:40.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:40.377 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:19:40.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:40.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:19:40.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:40.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:19:40.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:41.700 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:41.696+0000 7f1d2538c7c0 -1 osd.0 28 log_to_monitors true 2024-09-04T19:19:41.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:41.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:41.828 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:19:41.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:19:41.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:41.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:19:42.339 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/1430323642,v1:127.0.0.1:6803/1430323642] [v2:127.0.0.1:6804/1430323642,v1:127.0.0.1:6805/1430323642] exists,up 79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:19:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:19:42.343 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:19:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:19:42.345 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:19:42.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:19:42.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:19:42.346 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-04T19:19:42.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:19:42.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:19:42.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:19:42.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:19:42.388 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:42.384+0000 7f2db5b467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:42.403 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:42.400+0000 7f2db5b467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:42.405 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:42.400+0000 7f2db5b467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:42.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:19:42.754 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:19:42.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:42.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:19:43.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:43.470 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:43.464+0000 7f2db5b467c0 -1 Falling back to public interface 2024-09-04T19:19:44.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:44.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:44.203 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:19:44.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:19:44.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:44.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:19:44.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:45.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:45.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:45.671 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:19:45.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:19:45.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:19:45.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:46.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:19:46.678 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:19:46.672+0000 7f2db5b467c0 -1 osd.2 28 log_to_monitors true 2024-09-04T19:19:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:19:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:19:47.086 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:19:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:19:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:19:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:19:47.550 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3075480344,v1:127.0.0.1:6811/3075480344] [v2:127.0.0.1:6812/3075480344,v1:127.0.0.1:6813/3075480344] exists,up f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:19:47.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:19:47.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:19:47.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:19:47.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:47.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-04T19:19:48.082 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already in. 2024-09-04T19:19:48.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-04T19:19:48.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-04T19:19:48.968 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-04T19:19:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:19:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-04T19:19:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:19:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:19:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:19:48.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:19:49.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:19:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:19:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:19:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:19:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:19:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:49.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:19:49.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-09-04T19:19:49.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-09-04T19:19:49.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364' 2024-09-04T19:19:49.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:49.690 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:19:49.818 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:19:49.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:19:49.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:19:49.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:19:49.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:19:49.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789955 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789955 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855364 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:19:49.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:19:49.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855364 2024-09-04T19:19:49.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:19:49.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-09-04T19:19:49.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-09-04T19:19:49.984 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 150323855364 2024-09-04T19:19:49.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:19:50.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855364 2024-09-04T19:19:50.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:19:51.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:19:51.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:19:51.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-04T19:19:51.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:19:51.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789955 2024-09-04T19:19:51.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:19:51.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:19:51.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789955 2024-09-04T19:19:51.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:19:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789955 2024-09-04T19:19:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-09-04T19:19:51.863 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 158913789955 2024-09-04T19:19:51.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:19:52.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-04T19:19:52.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:19:52.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:52.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:52.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:19:52.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:52.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:52.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:52.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:52.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:52.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:52.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:53.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:19:53.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:53.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:53.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:19:53.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:19:53.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:19:53.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:53.912 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:53.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:53.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:53.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:53.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:53.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:54.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:19:54.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:54.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:54.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:54.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:19:54.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:19:54.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:19:54.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:19:54.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:19:54.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:19:54.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:19:54.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:19:54.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:19:55.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:19:55.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:19:55.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:19:55.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:19:55.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:19:55.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:19:55.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:55.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:55.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:55.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:55.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:55.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:55.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:56.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:19:56.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:56.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:56.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:19:56.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:19:56.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:19:56.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:19:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:19:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:19:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:19:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:19:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:19:57.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:19:57.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:57.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:57.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:57.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:57.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:57.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:57.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:19:58.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:19:58.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:19:58.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:19:58.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:19:58.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:19:58.537 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:19:58.537 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:19:59.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:19:59.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:19:59.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:19:59.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:19:59.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:19:59.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:19:59.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:19:59.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:19:59.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:19:59.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:19:59.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:19:59.976 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:19:59.976 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:20:00.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:20:00.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:20:00.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:20:00.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:20:00.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:20:00.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:20:01.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:20:01.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:20:01.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:20:01.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:20:01.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:20:03.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:20:03.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:20:03.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:20:03.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:20:03.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:20:03.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:20:03.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:20:03.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:20:03.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:20:03.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:20:03.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:20:03.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:20:04.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:20:04.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:20:04.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:20:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:20:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:20:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:20:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:20:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:20:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:20:07.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:20:07.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:20:07.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:20:07.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:20:07.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:20:07.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:20:07.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:20:08.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:20:08.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:20:08.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:20:08.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:20:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:20:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:20:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:20:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:20:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:20:08.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:20:08.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:20:08.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:20:08.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:20:09.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:20:09.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:20:09.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:20:09.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:20:09.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:20:15.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:20:15.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:20:15.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:20:16.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:20:16.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:20:16.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:20:16.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:20:16.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:20:16.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:20:17.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:20:17.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:20:17.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:20:17.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:20:17.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:20:24.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:20:24.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:20:24.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:20:24.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:20:24.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:20:24.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:20:24.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:20:25.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:20:25.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:20:25.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:20:25.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:20:25.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:20:25.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:20:25.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:20:25.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:20:25.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:20:26.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:20:26.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:20:26.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:20:26.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "217f4e09-78db-49e1-a538-a9f3e00a6c51", 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:20:26.690330+0000", 2024-09-04T19:20:26.693 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:20:26.694 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:20:26.695 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:20:26.696 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.697 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:17:02.510275Z", 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:17:02.510275Z", 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:20:26.698 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:20:26.699 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.700 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:20:26.701 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 42, 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:17:03.315876+0000", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:19:49.047788+0000", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:19:46.997434+0000", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:19:48.549541+0000", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:20:26.702 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 9, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:20:26.703 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:18:49.924908+0000", 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 1, 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool", 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:20:26.704 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 0, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "off", 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 1, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 1, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 1, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 1, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 1, 2024-09-04T19:20:26.705 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "28", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "selfmanaged", 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 0, 2024-09-04T19:20:26.706 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 0, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [], 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:20:26.707 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "", 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:20:26.708 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 0, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {}, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "read_balance": { 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "score_type": "Fair distribution", 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "score_acting": 2.9900000095367432, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "score_stable": 2.9900000095367432, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_score": 0.67000001668930054, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_acting": 2, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_stable": 2, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity_weighted": 1, 2024-09-04T19:20:26.709 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity": 1, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity_weighted": 1 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "79f12bb7-8ec8-4178-aacb-b507f8d310dc", 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:20:26.710 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 28, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 35, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 40, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 29, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.711 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.712 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.713 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.714 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1430323642 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/1430323642", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/1430323642", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1430323642", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1430323642", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:20:26.715 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "2c064751-aed6-4a4d-a8d9-0a69663e2801", 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 23, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 30, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:20:26.716 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.717 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:20:26.718 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:20:26.719 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2805168138 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/2805168138", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/2805168138", 2024-09-04T19:20:26.720 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2805168138", 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2805168138", 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "f23fe3fd-faa7-40fe-a7bd-945126d4d79f", 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 15, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 28, 2024-09-04T19:20:26.721 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 37, 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 0, 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 31, 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.722 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.723 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.724 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3075480344 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/3075480344", 2024-09-04T19:20:26.725 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/3075480344", 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3075480344", 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3075480344", 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:19:34.132438+0000", 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:20:26.726 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:17:36.459386+0000", 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:19:34.706463+0000", 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:18:10.733996+0000", 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.727 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:19:35.297997+0000", 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:18:45.608675+0000", 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:20:26.728 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:20:26.729 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:20:26.730 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/1430323642,v1:127.0.0.1:6805/1430323642]", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:20:26.731 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:17:09.999391Z", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:20:26.732 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/1430323642,v1:127.0.0.1:6803/1430323642]", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1430323642,v1:127.0.0.1:6809/1430323642]", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1430323642,v1:127.0.0.1:6807/1430323642]", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:20:26.733 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/2805168138,v1:127.0.0.1:6813/2805168138]", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:20:26.734 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:20:26.735 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:17:43.686585Z", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:20:26.736 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/2805168138,v1:127.0.0.1:6811/2805168138]", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2805168138,v1:127.0.0.1:6817/2805168138]", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2805168138,v1:127.0.0.1:6815/2805168138]", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:20:26.737 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/3075480344,v1:127.0.0.1:6813/3075480344]", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:20:26.738 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:20:26.739 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:18:18.651321Z", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:20:26.740 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/3075480344,v1:127.0.0.1:6811/3075480344]", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3075480344,v1:127.0.0.1:6817/3075480344]", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3075480344,v1:127.0.0.1:6815/3075480344]", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:20:26.741 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.742 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 35 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 37 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 42, 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:20:26.743 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:20:26.744 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.745 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.746 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.747 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.748 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.749 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:20:26.750 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:20:26.751 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:20:26.752 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.753 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:20:26.754 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.755 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:20:26.756 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 0, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:20:26.757 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:17:03:292047+0000", 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:20:26.758 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:20:26.759 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:20:26.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 1, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 61700, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:20:26.761 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 100, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 100, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:20:26.762 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:20:26.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:20:26.764 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 16, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 16, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:20:26.765 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83688, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2120, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489112, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036850688, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2170880, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 462038, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:20:26.766 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10395, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416933 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 1, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:20:26.767 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:20:26.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83688, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2120, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489112, 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:20:26.769 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036850688, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2170880, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 462038, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10395, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416933 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:20:26.770 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 1, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.771 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 61700, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:20:26.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 100, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 100, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:20:26.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:20:26.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1228800, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 185100, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:20:26.775 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 16, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 16, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 150323855371 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.776 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672984 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 158913789962 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 1 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.777 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 1, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.778 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 5, 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:19:36.845473+0000", 2024-09-04T19:20:26.779 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 90, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 379, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 519, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:20:26.780 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:20:26.781 INFO:tasks.workunit.client.0.smithi174.stderr:report 2550963156 2024-09-04T19:20:26.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:20:26.781 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-04T19:20:26.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-04T19:20:26.782 INFO:tasks.workunit.client.0.smithi174.stdout:writing non-divergent object existing_4 2024-09-04T19:20:26.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-04T19:20:26.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:27.093 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:27.094 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 28'100 42:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:27.094 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:27.094 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:27.094 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:27.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-04T19:20:27.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-04T19:20:27.234 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery 2024-09-04T19:20:27.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:27.623 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:27.623 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 28'100 42:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:27.623 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:27.623 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:27.623 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:27.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-04T19:20:27.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-04T19:20:27.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:20:27.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-04T19:20:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok set_recovery_delay 100000 2024-09-04T19:20:27.748 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:20:27.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-04T19:20:27.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-04T19:20:27.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:20:27.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:20:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:20:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:20:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:20:27.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:20:27.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-04T19:20:27.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok set_recovery_delay 100000 2024-09-04T19:20:27.877 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:20:27.891 INFO:tasks.workunit.client.0.smithi174.stdout:revive divergent 1 2024-09-04T19:20:27.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-04T19:20:27.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:28.272 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:28.273 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 28'100 42:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:28.273 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:28.273 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:28.273 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:28.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-04T19:20:29.034 INFO:tasks.workunit.client.0.smithi174.stderr:noup is set 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:20:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:20:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:20:29.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:20:29.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:20:29.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:20:29.056 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:20:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:20:29.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:20:29.058 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:20:29.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:20:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:20:29.099 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:29.095+0000 7fe588a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:29.120 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:29.115+0000 7fe588a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:29.121 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:29.115+0000 7fe588a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:29.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-04T19:20:29.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-04T19:20:30.378 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:30.371+0000 7fe588a887c0 -1 Falling back to public interface 2024-09-04T19:20:33.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:33.351+0000 7fe588a887c0 -1 osd.1 28 log_to_monitors true 2024-09-04T19:20:34.479 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery divergent 2024-09-04T19:20:34.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-04T19:20:34.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:34.888 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:34.888 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 42'101 44:133 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:34.888 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:34.888 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:34.889 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:34.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-04T19:20:34.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:20:34.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-04T19:20:34.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok set_recovery_delay 100000 2024-09-04T19:20:35.017 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:20:35.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-04T19:20:35.772 INFO:tasks.workunit.client.0.smithi174.stderr:noup is unset 2024-09-04T19:20:35.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-04T19:20:35.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:20:35.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:20:35.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:20:36.302 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 35 up_thru 40 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/1430323642,v1:127.0.0.1:6803/1430323642] [v2:127.0.0.1:6804/1430323642,v1:127.0.0.1:6805/1430323642] exists,up 79f12bb7-8ec8-4178-aacb-b507f8d310dc 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:20:36.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:20:36.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:20:36.721 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/4257535433,v1:127.0.0.1:6819/4257535433] [v2:127.0.0.1:6820/4257535433,v1:127.0.0.1:6821/4257535433] exists,up 2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:20:36.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:20:36.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:20:36.723 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:20:36.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:20:36.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:20:37.135 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3075480344,v1:127.0.0.1:6811/3075480344] [v2:127.0.0.1:6812/3075480344,v1:127.0.0.1:6813/3075480344] exists,up f23fe3fd-faa7-40fe-a7bd-945126d4d79f 2024-09-04T19:20:37.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:20:37.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:20:37.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:20:37.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:37.542 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:37.542 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 42'101 44:133 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:37.542 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:37.542 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:37.543 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:37.557 INFO:tasks.workunit.client.0.smithi174.stdout:wait for peering 2024-09-04T19:20:37.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-04T19:20:37.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:37.940 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:37.940 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 42'101 44:133 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:37.940 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:37.940 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:37.941 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:37.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-04T19:20:38.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-04T19:20:38.092 INFO:tasks.workunit.client.0.smithi174.stdout:killing divergent 1 2024-09-04T19:20:38.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:38.491 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:38.492 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 42'101 44:133 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:38.492 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:38.492 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:38.492 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:38.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-04T19:20:38.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:20:38.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:20:38.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:20:38.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:20:38.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:20:38.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:20:38.616 INFO:tasks.workunit.client.0.smithi174.stdout:reviving divergent 1 2024-09-04T19:20:38.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-04T19:20:38.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-04T19:20:39.000 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:20:39.000 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:19:49.342423+0000 42'101 44:133 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:39:14.706574+0000 0 0 2024-09-04T19:20:39.000 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:20:39.000 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:20:39.000 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:20:39.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:20:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:20:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:20:39.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:20:39.021 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:20:39.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:20:39.054 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:39.047+0000 7f153bf617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:39.056 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:39.051+0000 7f153bf617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:39.057 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:39.051+0000 7f153bf617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:20:39.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:20:39.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:20:39.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:20:39.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:20:39.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:20:39.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:20:39.460 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:20:39.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:20:39.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:20:39.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:20:39.971 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/4257535433,v1:127.0.0.1:6819/4257535433] [v2:127.0.0.1:6820/4257535433,v1:127.0.0.1:6821/4257535433] exists,up 2c064751-aed6-4a4d-a8d9-0a69663e2801 2024-09-04T19:20:39.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:20:39.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:20:39.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:20:39.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-04T19:20:40.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:40.287+0000 7f153bf617c0 -1 Falling back to public interface 2024-09-04T19:20:43.313 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:20:43.311+0000 7f153bf617c0 -1 osd.1 48 log_to_monitors true 2024-09-04T19:20:59.978 INFO:tasks.workunit.client.0.smithi174.stdout:allowing recovery 2024-09-04T19:20:59.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-04T19:20:59.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-04T19:21:00.386 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:21:00.387 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-04T19:20:44.754446+0000 48'102 52:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T06:13:53.752624+0000 0 0 2024-09-04T19:21:00.387 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:21:00.387 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:21:00.387 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:21:00.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-04T19:21:00.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-04T19:21:00.553 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:21:00.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-04T19:21:00.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-04T19:21:00.721 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:21:00.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-04T19:21:00.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-04T19:21:00.885 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:21:00.899 INFO:tasks.workunit.client.0.smithi174.stdout:reading divergent objects 2024-09-04T19:21:00.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-04T19:21:00.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-04T19:21:01.294 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:21:01.294 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-04T19:20:44.754446+0000 48'102 52:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:18:49.987799+0000 0'0 2024-09-04T19:18:49.987799+0000 0 0 periodic scrub scheduled @ 2024-09-06T06:13:53.752624+0000 0 0 2024-09-04T19:21:01.294 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:21:01.294 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:21:01.295 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:21:01.310 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-04T19:21:01.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-04T19:21:01.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:21:01.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-04T19:21:01.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:21:01.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-04T19:21:01.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:21:01.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-04T19:21:01.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:21:01.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-04T19:21:01.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-04T19:21:01.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-04T19:21:01.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:21:01.503 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4618.0:1 2024-09-04T19:19:22.614807+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:21:01.503 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4618.0:1 2024-09-04T19:19:22.614807+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:21:01.503 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:21:01.503 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-09-04T19:21:01.503 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-04T19:21:01.504 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:21:01.504 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2024-09-04T19:21:01.504 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4618.0:1 2024-09-04T19:19:22.614807+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:21:01.504 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:20:36.931+0000 7fe56ca12640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-04T19:21:01.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-04T19:21:01.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:21:01.508 INFO:tasks.workunit.client.0.smithi174.stdout:success 2024-09-04T19:21:01.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-04T19:21:01.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-04T19:21:01.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-04T19:21:01.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-04T19:21:02.003 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' does not exist 2024-09-04T19:21:02.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-04T19:21:02.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:21:02.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:21:02.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:21:02.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:21:02.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:21:07.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:21:07.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-04T19:21:07.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:21:07.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:21:07.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:21:07.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:21:07.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:21:07.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:21:07.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:21:07.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:21:07.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:21:07.346 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:21:07.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:21:07.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:21:07.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:21:07.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:21:07.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:21:07.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:21:07.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:21:07.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:21:07.350 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:21:07.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:21:07.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:21:07.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:21:07.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:21:07.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:07.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:07.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:21:07.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:21:07.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:21:07.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:21:07.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:21:07.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:21:07.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:21:07.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:21:07.383 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:21:07.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:21:07.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:21:07.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:21:07.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:21:07.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:21:07.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:21:07.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:21:07.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:21:07.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:21:07.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:21:07.389 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:21:07.389 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:07.389 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:07.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:21:07.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:21:07.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:21:07.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-04T19:21:07.391 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:21:07.391 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:07.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:07.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27364 2024-09-04T19:21:07.393 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:21:07.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-04T19:21:07.394 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-04T19:21:07.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-04T19:21:07.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-04T19:21:07.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:21:07.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:21:07.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:21:07.794 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:21:07.794 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:07.794 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:07.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:07.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:21:07.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:21:07.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:21:07.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:21:07.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:21:07.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:21:07.999 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:21:07.999 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:21:07.999 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:21:07.999 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:21:08.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get fsid 2024-09-04T19:21:08.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:08.117 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:08.118 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:21:08.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:21:08.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get mon_host 2024-09-04T19:21:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-04T19:21:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-04T19:21:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:21:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:21:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:21:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-04T19:21:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:21:09.141 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:09.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:21:09.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:21:09.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:09.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:21:09.184 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:21:09.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:21:09.185 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:21:09.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5057219b-c82e-4a4d-9b96-c5a119c90e41' 2024-09-04T19:21:09.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:21:09.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAls9hmJxxBDBAAjeAde1to6+oy4Acoj+TK9w== 2024-09-04T19:21:09.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAls9hmJxxBDBAAjeAde1to6+oy4Acoj+TK9w=="}' 2024-09-04T19:21:09.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5057219b-c82e-4a4d-9b96-c5a119c90e41 -i td/divergent-priors/0/new.json 2024-09-04T19:21:09.509 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:21:09.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-04T19:21:09.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAls9hmJxxBDBAAjeAde1to6+oy4Acoj+TK9w== --osd-uuid 5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:21:09.575 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:09.571+0000 7fd5e03a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:09.577 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:09.575+0000 7fd5e03a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:09.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:09.575+0000 7fd5e03a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:09.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:09.575+0000 7fd5e03a17c0 -1 bdev(0x557d92562a80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:21:09.579 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:09.575+0000 7fd5e03a17c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-04T19:21:15.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-04T19:21:15.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:21:15.488 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:21:15.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:21:15.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:21:16.071 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:21:16.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:21:16.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:21:16.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:21:16.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:21:16.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:21:16.111 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:16.111+0000 7fbb167727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:16.123 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:16.123+0000 7fbb167727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:16.124 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:16.123+0000 7fbb167727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:16.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:21:16.502 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:21:16.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:16.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:16.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:17.379 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:17.375+0000 7fbb167727c0 -1 Falling back to public interface 2024-09-04T19:21:17.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:17.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:17.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:21:17.973 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:21:17.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:17.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:18.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:19.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:19.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:19.489 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:21:19.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:21:19.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:19.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:19.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:20.113 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:20.111+0000 7fbb167727c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:21:20.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:20.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:20.923 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:21:20.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:21:20.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:20.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:21.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:22.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:22.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:22.453 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:21:22.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:21:22.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:22.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:22.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:23.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:23.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:23.934 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:21:23.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:21:23.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:23.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:24.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:25.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:25.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:25.391 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:21:25.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:21:25.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:25.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:25.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:26.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:26.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:26.809 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:21:26.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:21:26.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:26.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:27.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:28.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:28.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:28.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:21:28.284 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:21:28.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:28.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:28.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:29.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:29.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:29.756 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:21:29.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:21:29.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:29.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:30.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:31.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:31.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:31.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:21:31.256 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:21:31.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:31.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:31.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:32.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:32.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:32.883 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:21:32.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:21:32.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:32.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:33.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:34.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:34.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:34.366 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:21:34.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:21:34.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:34.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:35.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:36.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:36.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:36.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:21:36.163 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:21:36.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:36.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:37.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:38.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:38.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:38.110 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:21:38.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:21:38.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:38.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:38.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:39.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:39.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:39.927 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:21:39.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:21:39.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:39.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:40.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:41.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:41.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:41.857 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:21:41.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:21:41.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:41.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:21:42.348 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/307174194,v1:127.0.0.1:6803/307174194] [v2:127.0.0.1:6804/307174194,v1:127.0.0.1:6805/307174194] exists,up 5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-04T19:21:42.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:21:42.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:21:42.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:21:42.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:42.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:21:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:21:42.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:21:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:21:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b538fa63-4022-4fdf-8310-4533b70a9c58' 2024-09-04T19:21:42.355 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:21:42.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:21:42.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGs9hm22NiFhAA/AgwkEWkaWZz6BrA8suaqQ== 2024-09-04T19:21:42.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGs9hm22NiFhAA/AgwkEWkaWZz6BrA8suaqQ=="}' 2024-09-04T19:21:42.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b538fa63-4022-4fdf-8310-4533b70a9c58 -i td/divergent-priors/1/new.json 2024-09-04T19:21:42.868 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:21:42.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-04T19:21:42.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGs9hm22NiFhAA/AgwkEWkaWZz6BrA8suaqQ== --osd-uuid b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:21:42.934 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:42.930+0000 7f5f88aa27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:42.936 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:42.934+0000 7f5f88aa27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:42.937 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:42.934+0000 7f5f88aa27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:42.937 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:42.934+0000 7f5f88aa27c0 -1 bdev(0x563d4bd6ca80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:21:42.937 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:42.934+0000 7f5f88aa27c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-04T19:21:49.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-04T19:21:49.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:21:49.146 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:21:49.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:21:49.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:21:49.700 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:21:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:21:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:21:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:21:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:21:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:21:49.741 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:49.738+0000 7fed63f2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:49.744 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:49.742+0000 7fed63f2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:49.745 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:49.742+0000 7fed63f2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:21:50.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:21:50.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:21:50.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:21:50.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:21:50.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:21:50.182 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:21:50.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:50.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:21:50.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:50.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:50.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:51.242 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:51.238+0000 7fed63f2f7c0 -1 Falling back to public interface 2024-09-04T19:21:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:51.600 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:21:51.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:21:51.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:51.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:52.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:53.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:53.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:53.033 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:21:53.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:21:53.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:53.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:53.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:54.063 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:21:54.058+0000 7fed63f2f7c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:21:54.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:54.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:54.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:21:54.508 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:21:54.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:54.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:55.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:56.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:56.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:56.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:21:56.129 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:21:56.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:56.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:56.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:57.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:57.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:57.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:21:57.665 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:21:57.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:57.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:58.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:21:59.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:21:59.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:21:59.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:21:59.105 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:21:59.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:21:59.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:21:59.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:00.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:00.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:00.547 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:22:00.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:22:00.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:00.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:01.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:02.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:02.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:02.018 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:22:02.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:22:02.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:02.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:02.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:03.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:03.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:03.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:22:03.512 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:22:03.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:03.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:04.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:05.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:05.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:05.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:22:05.038 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:22:05.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:05.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:05.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:06.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:06.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:06.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:22:06.482 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:22:06.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:06.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:07.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:08.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:08.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:22:08.171 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:22:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:08.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:09.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:22:09.700 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:22:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:10.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:22:11.361 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:22:11.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:11.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:13.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:13.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:13.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:22:13.110 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:22:13.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:13.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:13.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:14.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:14.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:14.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:22:14.857 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:22:14.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:14.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:15.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:16.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:16.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:16.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:22:16.591 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:22:16.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:16.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:17.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:18.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:18.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:18.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:22:18.089 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:22:18.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:18.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:22:18.552 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/641658101,v1:127.0.0.1:6811/641658101] [v2:127.0.0.1:6812/641658101,v1:127.0.0.1:6813/641658101] exists,up b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:22:18.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:22:18.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:22:18.557 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:22:18.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:22:18.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:22:18.560 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:22:18.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 22cd6806-54c3-4f4b-aa24-f1086b7ea47e' 2024-09-04T19:22:18.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:22:18.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqs9hm1i7tIhAAx0cxYgopQ4jA3tVndNVuPQ== 2024-09-04T19:22:18.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqs9hm1i7tIhAAx0cxYgopQ4jA3tVndNVuPQ=="}' 2024-09-04T19:22:18.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22cd6806-54c3-4f4b-aa24-f1086b7ea47e -i td/divergent-priors/2/new.json 2024-09-04T19:22:19.072 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:22:19.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-04T19:22:19.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqs9hm1i7tIhAAx0cxYgopQ4jA3tVndNVuPQ== --osd-uuid 22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:22:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:19.138+0000 7fa279e857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:19.142 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:19.138+0000 7fa279e857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:19.143 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:19.142+0000 7fa279e857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:19.144 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:19.142+0000 7fa279e857c0 -1 bdev(0x56473fb0aa80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:22:19.144 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:19.142+0000 7fa279e857c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-04T19:22:25.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-04T19:22:25.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:22:25.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:22:25.129 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:22:25.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:22:25.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:22:25.710 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:22:25.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:22:25.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:22:25.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:22:25.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:22:25.750 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:25.746+0000 7f4db68aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:25.757 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:25.754+0000 7f4db68aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:25.758 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:25.758+0000 7f4db68aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:22:26.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:26.143 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:22:26.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:22:26.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:26.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:26.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:26.801 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:26.798+0000 7f4db68aa7c0 -1 Falling back to public interface 2024-09-04T19:22:27.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:27.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:27.550 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:22:27.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:22:27.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:27.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:27.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:22:28.990 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:22:28.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:28.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:29.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:29.511 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:22:29.510+0000 7f4db68aa7c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:22:30.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:30.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:30.409 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:22:30.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:22:30.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:30.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:30.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:31.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:31.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:31.863 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:22:31.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:22:31.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:31.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:32.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:33.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:33.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:22:33.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:22:33.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:33.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:33.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:34.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:34.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:34.903 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:22:34.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:22:34.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:34.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:35.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:36.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:36.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:36.461 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:22:36.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:22:36.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:36.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:36.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:37.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:37.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:37.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:22:37.926 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:22:37.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:37.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:38.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:39.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:39.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:39.405 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:22:39.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:22:39.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:39.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:39.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:40.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:40.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:40.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:22:40.923 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:22:40.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:40.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:41.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:42.407 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:22:42.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:22:42.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:42.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:43.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:44.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:44.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:44.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:22:44.544 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:22:44.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:44.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:45.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:46.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:46.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:46.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:22:46.636 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:22:46.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:46.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:47.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:48.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:48.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:48.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:22:48.586 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:22:48.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:48.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:49.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:50.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:50.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:50.381 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:22:50.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:22:50.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:50.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:51.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:22:52.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:22:52.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:22:52.328 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:22:52.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:22:52.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:22:52.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:22:52.755 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2705655067,v1:127.0.0.1:6819/2705655067] [v2:127.0.0.1:6820/2705655067,v1:127.0.0.1:6821/2705655067] exists,up 22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:22:52.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:22:52.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:22:52.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:22:52.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-04T19:22:53.391 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-04T19:22:53.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-04T19:22:54.119 INFO:tasks.workunit.client.0.smithi174.stderr:noin is set 2024-09-04T19:22:54.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-04T19:22:55.015 INFO:tasks.workunit.client.0.smithi174.stderr:nodown is set 2024-09-04T19:22:55.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-04T19:22:55.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-04T19:22:55.796 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' already exists 2024-09-04T19:22:55.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:22:56.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-04T19:22:57.452 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 size to 3 2024-09-04T19:22:57.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-04T19:22:58.406 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 min_size to 2 2024-09-04T19:22:58.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-04T19:22:58.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:22:58.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:22:58.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:22:58.911 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:22:58.911 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:22:58.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:22:58.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:22:58.912 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:22:59.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-09-04T19:22:59.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-09-04T19:22:59.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-09-04T19:22:59.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:22:59.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:22:59.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-09-04T19:22:59.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-09-04T19:22:59.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672976' 2024-09-04T19:22:59.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:22:59.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:22:59.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-09-04T19:22:59.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-09-04T19:22:59.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672976 2-64424509449' 2024-09-04T19:22:59.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:22:59.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-09-04T19:22:59.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:22:59.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:22:59.410 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-09-04T19:22:59.410 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:22:59.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-09-04T19:22:59.411 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836501 2024-09-04T19:22:59.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-04T19:22:59.412 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:22:59.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-04T19:22:59.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:22:59.887 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:22:59.887 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-09-04T19:22:59.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:22:59.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-09-04T19:22:59.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:22:59.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-09-04T19:22:59.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-04T19:22:59.892 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672976 2024-09-04T19:22:59.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:23:00.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-04T19:23:00.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:00.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-09-04T19:23:00.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:00.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:23:00.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-09-04T19:23:00.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:00.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-09-04T19:23:00.360 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509449 2024-09-04T19:23:00.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-04T19:23:00.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:23:00.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:23:00.834 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:23:00.834 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:23:00.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:23:00.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:23:00.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:23:00.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:23:00.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:23:00.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:23:00.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:23:00.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:23:00.934 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:23:01.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:23:01.357 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:23:01.357 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:23:01.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:23:01.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:01.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:23:01.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-09-04T19:23:01.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-09-04T19:23:01.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-09-04T19:23:01.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:01.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:23:01.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-09-04T19:23:01.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-09-04T19:23:01.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-09-04T19:23:01.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:01.679 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:23:01.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-09-04T19:23:01.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-09-04T19:23:01.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509451' 2024-09-04T19:23:01.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:01.832 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-09-04T19:23:01.832 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:01.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:23:01.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-09-04T19:23:01.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:01.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-09-04T19:23:01.835 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836503 2024-09-04T19:23:01.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-04T19:23:01.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:23:02.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-04T19:23:02.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:02.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:02.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-09-04T19:23:02.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:23:02.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-09-04T19:23:02.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:02.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-09-04T19:23:02.233 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672978 2024-09-04T19:23:02.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-04T19:23:02.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:23:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-04T19:23:02.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:02.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-09-04T19:23:02.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:02.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:23:02.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-09-04T19:23:02.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:02.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-09-04T19:23:02.717 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509451 2024-09-04T19:23:02.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-04T19:23:02.718 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:03.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509451 2024-09-04T19:23:03.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:23:04.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:23:04.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:04.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-04T19:23:04.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:23:04.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:04.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:23:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:23:05.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:23:05.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:23:05.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:23:05.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:23:05.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:23:05.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:23:05.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:23:05.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:23:05.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:05.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:06.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:23:06.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:23:06.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:23:06.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-04T19:23:06.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-04T19:23:06.444 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:23:06.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-04T19:23:06.451 INFO:tasks.workunit.client.0.smithi174.stdout:primary and soon to be divergent is 1 2024-09-04T19:23:06.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-04T19:23:06.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:23:06.852 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:23:06.853 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-04T19:22:56.357847+0000 0'0 29:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:00:19.033779+0000 0 0 2024-09-04T19:23:06.853 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:23:06.853 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:23:06.853 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-04T19:23:06.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-04T19:23:06.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-04T19:23:06.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-04T19:23:06.869 INFO:tasks.workunit.client.0.smithi174.stdout:writing initial objects 2024-09-04T19:23:06.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-04T19:23:06.869 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-04T19:23:06.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:06.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-04T19:23:07.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-04T19:23:07.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-04T19:23:07.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-04T19:23:07.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-04T19:23:07.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-04T19:23:07.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-04T19:23:07.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:07.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-04T19:23:08.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-04T19:23:08.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-04T19:23:08.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-04T19:23:08.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-04T19:23:08.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-04T19:23:08.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:08.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-04T19:23:09.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-04T19:23:09.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-04T19:23:09.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-04T19:23:09.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-04T19:23:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-04T19:23:09.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:09.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-04T19:23:10.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-04T19:23:10.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-04T19:23:10.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-04T19:23:10.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-04T19:23:10.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-04T19:23:10.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-04T19:23:10.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-04T19:23:11.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-04T19:23:11.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-04T19:23:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-04T19:23:11.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-04T19:23:11.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-04T19:23:11.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:11.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-04T19:23:12.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-04T19:23:12.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-04T19:23:12.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-04T19:23:12.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-04T19:23:12.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-04T19:23:12.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:12.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-04T19:23:13.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-04T19:23:13.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-04T19:23:13.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-04T19:23:13.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-04T19:23:13.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-04T19:23:13.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:13.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-04T19:23:14.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-04T19:23:14.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-04T19:23:14.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-04T19:23:14.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-04T19:23:14.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-04T19:23:14.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:14.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-04T19:23:15.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-04T19:23:15.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-04T19:23:15.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-04T19:23:15.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-04T19:23:15.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-04T19:23:15.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-04T19:23:15.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:15.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-04T19:23:16.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-04T19:23:16.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-04T19:23:16.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-04T19:23:16.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-04T19:23:16.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-04T19:23:16.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:16.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-04T19:23:17.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-04T19:23:17.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-04T19:23:17.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-04T19:23:17.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-04T19:23:17.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-04T19:23:17.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:17.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-04T19:23:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-04T19:23:18.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-04T19:23:18.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-04T19:23:18.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-04T19:23:18.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-04T19:23:18.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-04T19:23:18.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:18.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-04T19:23:19.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-04T19:23:19.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-04T19:23:19.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-04T19:23:19.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-04T19:23:19.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-04T19:23:19.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:19.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-04T19:23:20.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-04T19:23:20.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-04T19:23:20.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-04T19:23:20.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-04T19:23:20.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-04T19:23:20.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:20.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-04T19:23:21.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-04T19:23:21.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-04T19:23:21.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-04T19:23:21.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-04T19:23:21.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-04T19:23:21.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-04T19:23:21.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:21.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-04T19:23:22.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:22.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-04T19:23:22.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:22.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-04T19:23:22.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:22.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-04T19:23:22.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-04T19:23:22.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-04T19:23:22.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:23:22.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-04T19:23:22.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:23:22.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:23:22.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:23:22.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:23:22.735 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:23:22.735 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:23:22.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:23:22.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:23:22.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:23:22.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:23:22.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:23:22.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:23:22.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:23:22.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:23:22.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:23.268 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:23:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-09-04T19:23:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-09-04T19:23:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-09-04T19:23:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:23.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:23:23.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-09-04T19:23:23.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-09-04T19:23:23.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-09-04T19:23:23.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:23.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:23:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-09-04T19:23:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-09-04T19:23:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509458' 2024-09-04T19:23:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:23.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-09-04T19:23:23.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:23.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:23:23.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-09-04T19:23:23.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:23.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-09-04T19:23:23.770 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836510 2024-09-04T19:23:23.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-04T19:23:23.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:23:24.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-04T19:23:24.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:24.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-09-04T19:23:24.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:24.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:23:24.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-09-04T19:23:24.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-09-04T19:23:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-04T19:23:24.184 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672985 2024-09-04T19:23:24.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:23:24.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-04T19:23:24.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:24.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-09-04T19:23:24.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:24.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:23:24.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-09-04T19:23:24.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:24.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-09-04T19:23:24.743 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509458 2024-09-04T19:23:24.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-04T19:23:24.744 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:25.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509458 2024-09-04T19:23:25.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:23:26.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:23:26.181 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:26.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-04T19:23:26.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:23:26.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:26.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:23:27.293 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:23:27.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:23:27.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:23:27.685 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:23:27.685 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:27.685 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:28.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:23:28.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:23:28.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:23:28.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-04T19:23:28.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-04T19:23:28.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-04T19:23:28.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-04T19:23:28.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-04T19:23:28.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-04T19:23:28.661 INFO:tasks.workunit.client.0.smithi174.stdout:blackholing osds 0 2 2024-09-04T19:23:28.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-04T19:23:28.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:23:29.057 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:23:29.057 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-04T19:22:56.357847+0000 29'100 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:00:19.033779+0000 0 0 2024-09-04T19:23:29.057 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:23:29.057 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:23:29.057 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:23:29.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:29.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:23:29.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:23:29.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:23:29.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-04T19:23:29.190 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:23:29.190 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:23:29.190 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:23:29.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:29.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-04T19:23:29.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:23:29.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:23:29.204 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:23:29.205 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:23:29.205 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:23:29.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:23:29.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:23:29.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-04T19:23:29.313 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:23:29.313 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:23:29.313 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:23:29.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-04T19:23:29.324 INFO:tasks.workunit.client.0.smithi174.stdout:create new divergent objects 2024-09-04T19:23:29.325 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-04T19:23:29.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:23:29.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:23:29.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-04T19:23:29.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-04T19:23:29.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-04T19:23:29.328 INFO:tasks.workunit.client.0.smithi174.stdout:writing divergent objects 2024-09-04T19:23:29.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-04T19:23:29.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-04T19:23:29.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-04T19:23:29.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-04T19:23:29.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-04T19:23:29.330 INFO:tasks.workunit.client.0.smithi174.stdout:remove divergent objects 2024-09-04T19:23:29.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-04T19:23:29.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-04T19:23:29.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-04T19:23:29.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-04T19:23:29.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-04T19:23:29.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-04T19:23:29.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-04T19:23:29.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-04T19:23:29.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-04T19:23:29.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-04T19:23:29.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-04T19:23:39.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-04T19:23:39.344 INFO:tasks.workunit.client.0.smithi174.stderr:rados: no process found 2024-09-04T19:23:39.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-04T19:23:39.344 INFO:tasks.workunit.client.0.smithi174.stdout:killing all the osds 2024-09-04T19:23:39.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:23:39.750 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:23:39.750 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-09-04T19:22:56.357847+0000 29'106 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:00:19.033779+0000 0 0 2024-09-04T19:23:39.750 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:23:39.750 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:23:39.750 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:23:39.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-04T19:23:39.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:23:39.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:23:39.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:23:39.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:23:39.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:23:39.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:23:39.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-04T19:23:39.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-04T19:23:40.368 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:23:40.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-04T19:23:40.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-04T19:23:40.975 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:23:40.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-04T19:23:40.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-04T19:23:41.527 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:23:41.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:41.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-04T19:23:42.166 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-04T19:23:42.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:42.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-04T19:23:42.813 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-04T19:23:42.830 INFO:tasks.workunit.client.0.smithi174.stdout:bringing up non_divergent 0 2 2024-09-04T19:23:42.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-04T19:23:42.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:23:43.226 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:23:43.227 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-09-04T19:22:56.357847+0000 29'106 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:00:19.033779+0000 0 0 2024-09-04T19:23:43.227 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:23:43.227 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:23:43.227 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:23:43.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:23:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:23:43.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:23:43.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:23:43.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:23:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:23:43.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:23:43.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:23:43.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:23:43.249 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:23:43.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:23:43.286 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:43.285+0000 7f62656707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:43.289 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:43.285+0000 7f62656707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:43.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:43.285+0000 7f62656707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:23:43.692 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:23:43.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:43.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:23:44.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:44.537 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:44.533+0000 7f62656707c0 -1 Falling back to public interface 2024-09-04T19:23:45.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:45.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:45.161 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:23:45.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:23:45.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:45.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:23:45.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:46.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:46.600 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:23:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:23:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:46.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:23:47.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:47.967 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:47.965+0000 7f62656707c0 -1 osd.0 29 log_to_monitors true 2024-09-04T19:23:48.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:48.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:48.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:23:48.013 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:23:48.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:48.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:23:48.601 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1313135797,v1:127.0.0.1:6803/1313135797] [v2:127.0.0.1:6804/1313135797,v1:127.0.0.1:6805/1313135797] exists,up 5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:23:48.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:23:48.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:23:48.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:23:48.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:23:48.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:23:48.608 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:23:48.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:23:48.609 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-04T19:23:48.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:23:48.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:23:48.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:23:48.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:23:48.640 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:48.637+0000 7fac550667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:48.666 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:48.665+0000 7fac550667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:48.668 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:48.665+0000 7fac550667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:23:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:23:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:23:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:23:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:23:49.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:23:49.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:49.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:23:49.055 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:23:49.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:49.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:23:49.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:49.536 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:49.533+0000 7fac550667c0 -1 Falling back to public interface 2024-09-04T19:23:50.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:50.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:50.507 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:23:50.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:23:50.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:50.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:23:50.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:51.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:51.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:51.925 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:23:51.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:23:51.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:51.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:23:52.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:23:52.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:23:52.737+0000 7fac550667c0 -1 osd.2 29 log_to_monitors true 2024-09-04T19:23:53.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:23:53.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:23:53.364 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:23:53.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:23:53.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:23:53.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:23:53.848 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/154937205,v1:127.0.0.1:6811/154937205] [v2:127.0.0.1:6812/154937205,v1:127.0.0.1:6813/154937205] exists,up 22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:23:53.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:23:53.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:23:53.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:23:53.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:53.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-04T19:23:54.370 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already in. 2024-09-04T19:23:54.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:23:54.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-04T19:23:54.898 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:23:54.914 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:23:54.915 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:23:54.915 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:23:54.915 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:23:54.915 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:23:54.915 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:23:54.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:23:55.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:23:55.418 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:23:55.419 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:23:55.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:23:55.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:55.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:23:55.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822661 2024-09-04T19:23:55.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822661 2024-09-04T19:23:55.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822661' 2024-09-04T19:23:55.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:55.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:23:55.702 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:23:55.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:23:55.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:23:55.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:23:55.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:23:55.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:23:55.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757252 2024-09-04T19:23:55.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757252 2024-09-04T19:23:55.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822661 2-163208757252' 2024-09-04T19:23:55.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:55.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-154618822661 2024-09-04T19:23:55.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:55.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:23:55.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-154618822661 2024-09-04T19:23:55.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:55.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822661 2024-09-04T19:23:55.867 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 154618822661 2024-09-04T19:23:55.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2024-09-04T19:23:55.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:23:56.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822661 -lt 154618822661 2024-09-04T19:23:56.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:23:56.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-163208757252 2024-09-04T19:23:56.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:23:56.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:23:56.277 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-163208757252 2024-09-04T19:23:56.277 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:23:56.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757252 2024-09-04T19:23:56.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2024-09-04T19:23:56.278 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 163208757252 2024-09-04T19:23:56.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:56.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757252 2024-09-04T19:23:56.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:23:57.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:23:57.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:23:58.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757252 -lt 163208757252 2024-09-04T19:23:58.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:23:58.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:58.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:23:58.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:23:59.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:23:59.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:23:59.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:23:59.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:23:59.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:23:59.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:23:59.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:23:59.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:23:59.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:23:59.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:23:59.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:23:59.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:23:59.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:00.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:00.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:00.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:00.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:00.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:01.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:01.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:01.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:24:01.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:01.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:24:01.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:01.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:01.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:01.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:01.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:01.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:01.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:02.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:02.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:02.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:02.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:24:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:24:03.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:03.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:03.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:03.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:03.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:03.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:03.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:04.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:04.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:04.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:24:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:24:05.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:05.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:05.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:05.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:05.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:05.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:05.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:05.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:06.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:06.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:06.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:06.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:06.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:06.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:06.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:07.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:07.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:07.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:24:07.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:07.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:24:08.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:08.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:08.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:08.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:08.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:08.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:08.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:08.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:09.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:09.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:09.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:09.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:09.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:09.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:09.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:09.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:10.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:10.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:10.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:24:10.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:10.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:24:13.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:13.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:13.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:13.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:13.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:13.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:13.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:13.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:13.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:14.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:14.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:14.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:15.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:15.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:15.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:24:15.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:15.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:21.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:21.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:21.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:21.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:21.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:21.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:21.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:22.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:22.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:22.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:22.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:22.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:22.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:22.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:22.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:22.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:23.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:23.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:23.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:24:23.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:23.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:24:30.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:30.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:30.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:30.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:30.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:30.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:30.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:31.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:31.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:31.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:31.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:31.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:31.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:31.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:31.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:31.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:31.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:31.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:31.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:24:31.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:24:32.385 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:24:32.385 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "111acff6-0466-407a-bec4-e5c0910dd53a", 2024-09-04T19:24:32.385 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:24:32.383556+0000", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.386 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.387 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:24:32.388 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:24:32.389 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:24:32.390 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:21:07.425984Z", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:21:07.425984Z", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:24:32.391 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:24:32.392 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:24:32.393 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 41, 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:24:32.394 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:21:08.288502+0000", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:23:55.168600+0000", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:23:52.999576+0000", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:23:54.742400+0000", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:24:32.395 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 9, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.396 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:22:55.364737+0000", 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 1, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool", 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 0, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:24:32.397 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "off", 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 1, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 1, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 1, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 1, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 1, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:24:32.398 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "29", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "selfmanaged", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 0, 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 0, 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [], 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:24:32.399 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:24:32.400 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "", 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:24:32.401 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 0, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {}, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "read_balance": { 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "score_type": "Fair distribution", 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "score_acting": 2.9900000095367432, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "score_stable": 2.9900000095367432, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_score": 0.67000001668930054, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_acting": 2, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_stable": 2, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity_weighted": 1, 2024-09-04T19:24:32.402 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity": 1, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity_weighted": 1 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "5057219b-c82e-4a4d-9b96-c5a119c90e41", 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:24:32.403 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 36, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 40, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 30, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.404 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.406 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.407 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/1313135797", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/1313135797", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1313135797", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1313135797", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.408 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "b538fa63-4022-4fdf-8310-4533b70a9c58", 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 23, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 31, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:24:32.409 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.410 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:24:32.411 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.412 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.413 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/641658101", 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/641658101", 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/641658101", 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/641658101", 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "22cd6806-54c3-4f4b-aa24-f1086b7ea47e", 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:24:32.414 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 15, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 38, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 0, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 32, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.415 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.416 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.417 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.418 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/154937205", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/154937205", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/154937205", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/154937205", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:24:32.419 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:40.227121+0000", 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:21:41.859694+0000", 2024-09-04T19:24:32.420 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:40.802087+0000", 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:22:16.734566+0000", 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.421 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:41.355363+0000", 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:22:51.788758+0000", 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:24:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:24:32.423 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:24:32.424 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/1313135797,v1:127.0.0.1:6805/1313135797]", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:24:32.425 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:21:14.845299Z", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:24:32.426 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/1313135797,v1:127.0.0.1:6803/1313135797]", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1313135797,v1:127.0.0.1:6809/1313135797]", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1313135797,v1:127.0.0.1:6807/1313135797]", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:24:32.427 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/641658101,v1:127.0.0.1:6813/641658101]", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:24:32.428 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:24:32.429 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:21:48.459063Z", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/641658101,v1:127.0.0.1:6811/641658101]", 2024-09-04T19:24:32.430 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/641658101,v1:127.0.0.1:6817/641658101]", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/641658101,v1:127.0.0.1:6815/641658101]", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:24:32.431 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/154937205,v1:127.0.0.1:6813/154937205]", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:24:32.432 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:24:32.433 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:22:24.471956Z", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/154937205,v1:127.0.0.1:6811/154937205]", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/154937205,v1:127.0.0.1:6817/154937205]", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/154937205,v1:127.0.0.1:6815/154937205]", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:24:32.434 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 36 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.436 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 38 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 41, 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:24:32.437 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:24:32.439 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:24:32.440 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.441 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:24:32.442 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:24:32.443 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.444 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.445 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:24:32.446 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:24:32.447 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.448 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:24:32.449 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 0, 2024-09-04T19:24:32.450 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:21:08:264224+0000", 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:24:32.451 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stderr:report 2268071353 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:24:32.452 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:24:32.453 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 1, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:24:32.454 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 61700, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:24:32.455 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 100, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 100, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:24:32.456 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:24:32.457 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:24:32.458 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 16, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 16, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:24:32.459 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83656, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2104, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489144, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036883456, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2154496, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 459927, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:24:32.460 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.461 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:24:32.462 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83656, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2104, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489144, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036883456, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2154496, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 459927, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:24:32.463 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.464 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:24:32.465 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 61700, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 100, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 100, 2024-09-04T19:24:32.466 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:24:32.467 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:24:32.468 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1220608, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 184566, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 16, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 16, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.469 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 154618822668 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672988 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 163208757259 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.470 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 1 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.471 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 1, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:24:32.472 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 5, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:23:43.689957+0000", 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 91, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 376, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 515, 2024-09-04T19:24:32.473 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-04T19:24:32.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-04T19:24:32.475 INFO:tasks.workunit.client.0.smithi174.stdout:writing non-divergent object existing_4 2024-09-04T19:24:32.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:24:32.790 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:24:32.790 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:24:32.790 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:24:32.790 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:24:32.790 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:24:32.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-04T19:24:32.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:24:32.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-04T19:24:32.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:24:32.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:24:32.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:24:32.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:24:32.919 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:24:32.919 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:24:32.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:24:32.920 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:24:32.920 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:24:32.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:24:33.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:24:33.418 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:24:33.419 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:24:33.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:24:33.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:24:33.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:24:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822670 2024-09-04T19:24:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822670 2024-09-04T19:24:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822670' 2024-09-04T19:24:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:24:33.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:24:33.692 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:24:33.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:24:33.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:24:33.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:24:33.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:24:33.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:24:33.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757261 2024-09-04T19:24:33.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757261 2024-09-04T19:24:33.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-09-04T19:24:33.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:24:33.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-154618822670 2024-09-04T19:24:33.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:24:33.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:24:33.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-154618822670 2024-09-04T19:24:33.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:24:33.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822670 2024-09-04T19:24:33.850 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 154618822670 2024-09-04T19:24:33.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-09-04T19:24:33.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:24:34.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-04T19:24:34.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:24:34.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-163208757261 2024-09-04T19:24:34.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:24:34.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:24:34.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-163208757261 2024-09-04T19:24:34.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:24:34.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757261 2024-09-04T19:24:34.267 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 163208757261 2024-09-04T19:24:34.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-04T19:24:34.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:24:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757259 -lt 163208757261 2024-09-04T19:24:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:24:35.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:24:35.718 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:24:36.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757262 -lt 163208757261 2024-09-04T19:24:36.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:24:36.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:36.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:36.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:24:36.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:36.730 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:37.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:37.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:37.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:37.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:37.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:37.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:24:37.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:24:37.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:24:37.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:37.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:24:37.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:37.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:37.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:37.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:37.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:37.749 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:37.749 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:37.749 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:38.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:38.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:38.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:38.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:38.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:38.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:39.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:39.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:39.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:24:39.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:39.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:24:39.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:39.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:39.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:39.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:39.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:39.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:39.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:39.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:39.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:39.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:39.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:39.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:40.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:40.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:40.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:41.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:41.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:41.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:24:41.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:41.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:24:41.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:41.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:41.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:41.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:41.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:41.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:41.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:41.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:41.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:41.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:41.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:41.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:42.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:42.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:42.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:42.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:42.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:42.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:42.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:42.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:24:42.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:42.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:24:43.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:43.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:43.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:43.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:43.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:43.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:43.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:43.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:44.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:44.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:44.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:44.129 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:44.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:44.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:44.655 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:44.655 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:45.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:45.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:45.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:24:45.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:45.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:24:46.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:46.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:46.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:46.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:46.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:46.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:46.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:46.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:47.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:47.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:47.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:47.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:47.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:47.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:48.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:48.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:48.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:24:48.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:48.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:24:51.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:51.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:51.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:51.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:51.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:51.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:51.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:51.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:51.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:51.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:51.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:51.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:24:52.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:24:52.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:24:52.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:24:52.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:24:53.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:24:53.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:24:53.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:24:53.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:24:53.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:24:59.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:24:59.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:24:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:24:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:24:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:24:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:24:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:24:59.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:24:59.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:24:59.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:24:59.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:24:59.911 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:25:00.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:25:00.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:25:00.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:25:00.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:25:00.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:25:00.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:25:00.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:25:00.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:25:00.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:25:01.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:25:01.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:25:01.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:25:01.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:25:01.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:25:08.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:25:08.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:25:08.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:25:08.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:25:08.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:25:08.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:25:08.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:25:08.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:25:08.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:25:08.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:25:08.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:25:08.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:25:09.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:25:09.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:25:09.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:25:09.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:25:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:25:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:25:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "111acff6-0466-407a-bec4-e5c0910dd53a", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:25:10.348462+0000", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:25:10.350 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:25:10.351 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 2 2024-09-04T19:25:10.352 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.353 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:25:10.354 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:21:07.425984Z", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:21:07.425984Z", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:25:10.355 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:25:10.356 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:25:10.357 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 41, 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:21:08.288502+0000", 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:23:55.168600+0000", 2024-09-04T19:25:10.358 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:23:52.999576+0000", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:23:54.742400+0000", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 9, 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:25:10.359 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:22:55.364737+0000", 2024-09-04T19:25:10.360 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 1, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool", 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 0, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "off", 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 1, 2024-09-04T19:25:10.361 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 1, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 1, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 1, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 1, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "29", 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:25:10.362 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "selfmanaged", 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 0, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 0, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [], 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:25:10.363 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "", 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:25:10.364 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 0, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {}, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "read_balance": { 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "score_type": "Fair distribution", 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "score_acting": 2.9900000095367432, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "score_stable": 2.9900000095367432, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_score": 0.67000001668930054, 2024-09-04T19:25:10.365 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_acting": 2, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_stable": 2, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity_weighted": 1, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity": 1, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity_weighted": 1 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "5057219b-c82e-4a4d-9b96-c5a119c90e41", 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:25:10.366 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 36, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 40, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 30, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.367 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.368 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.369 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.370 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 1313135797 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/1313135797", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/1313135797", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1313135797", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1313135797", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.371 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "b538fa63-4022-4fdf-8310-4533b70a9c58", 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 23, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 31, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:25:10.372 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.373 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.374 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.375 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 641658101 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.376 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/641658101", 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/641658101", 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/641658101", 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/641658101", 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "22cd6806-54c3-4f4b-aa24-f1086b7ea47e", 2024-09-04T19:25:10.377 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 15, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 38, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 0, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 32, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:25:10.378 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.379 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.380 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:25:10.381 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 154937205 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/154937205", 2024-09-04T19:25:10.382 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/154937205", 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/154937205", 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/154937205", 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:40.227121+0000", 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:25:10.383 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:21:41.859694+0000", 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:40.802087+0000", 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:22:16.734566+0000", 2024-09-04T19:25:10.384 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:23:41.355363+0000", 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:22:51.788758+0000", 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.385 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.386 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.387 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/1313135797,v1:127.0.0.1:6805/1313135797]", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:25:10.388 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:21:14.845299Z", 2024-09-04T19:25:10.389 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/1313135797,v1:127.0.0.1:6803/1313135797]", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1313135797,v1:127.0.0.1:6809/1313135797]", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1313135797,v1:127.0.0.1:6807/1313135797]", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:25:10.390 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/641658101,v1:127.0.0.1:6813/641658101]", 2024-09-04T19:25:10.391 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:25:10.392 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:25:10.393 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:21:48.459063Z", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/641658101,v1:127.0.0.1:6811/641658101]", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/641658101,v1:127.0.0.1:6817/641658101]", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/641658101,v1:127.0.0.1:6815/641658101]", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:25:10.394 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:25:10.395 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/154937205,v1:127.0.0.1:6813/154937205]", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:25:10.396 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:25:10.397 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:22:24.471956Z", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/154937205,v1:127.0.0.1:6811/154937205]", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/154937205,v1:127.0.0.1:6817/154937205]", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/154937205,v1:127.0.0.1:6815/154937205]", 2024-09-04T19:25:10.398 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:25:10.399 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 36 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:25:10.400 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 38 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 41, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:25:10.401 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:25:10.402 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:25:10.403 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:25:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.405 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:25:10.406 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:25:10.407 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:25:10.408 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:25:10.409 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:25:10.410 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:25:10.411 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:25:10.412 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.413 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:25:10.414 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 0, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.415 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:25:10.416 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:25:10.416 INFO:tasks.workunit.client.0.smithi174.stderr:report 1100311135 2024-09-04T19:25:10.416 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:21:08:264224+0000", 2024-09-04T19:25:10.416 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:25:10.416 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:25:10.417 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:25:10.418 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 1, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62050, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:25:10.419 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:25:10.420 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:25:10.421 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:25:10.422 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 17, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 17, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:25:10.423 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83656, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2104, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489144, 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:25:10.424 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036883456, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2154496, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 460627, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:25:10.425 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:25:10.426 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83656, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2104, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:25:10.427 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489144, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036883456, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2154496, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 460627, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:25:10.428 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:25:10.429 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:25:10.430 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62050, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 100, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 300, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 100, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 100, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:25:10.431 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:25:10.432 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.433 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1220608, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 185266, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 17, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 17, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 2, 2024-09-04T19:25:10.434 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 2, 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 154618822678 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672988 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 163208757269 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 1 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:25:10.436 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 1, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:25:10.437 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 7, 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:24:55.708122+0000", 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:25:10.438 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 110, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 439, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 597, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:25:10.439 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:25:10.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:25:10.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-04T19:25:10.440 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery 2024-09-04T19:25:10.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:10.765 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:10.765 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:10.765 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:10.765 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:10.766 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:10.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:25:10.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-04T19:25:10.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:25:10.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:25:10.784 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:10.784 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:10.784 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:10.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:25:10.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:25:10.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok set_recovery_delay 100000 2024-09-04T19:25:10.893 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:25:10.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:10.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:25:10.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:25:10.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok set_recovery_delay 100000 2024-09-04T19:25:11.020 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:25:11.034 INFO:tasks.workunit.client.0.smithi174.stdout:revive divergent 1 2024-09-04T19:25:11.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-04T19:25:11.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:11.439 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:11.439 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:11.439 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:11.439 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:11.439 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:11.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-04T19:25:12.121 INFO:tasks.workunit.client.0.smithi174.stderr:noup is set 2024-09-04T19:25:12.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-04T19:25:12.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:25:12.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:25:12.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:25:12.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:25:12.144 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:12.145 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:12.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:25:12.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:25:12.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:25:12.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:25:12.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:25:12.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:25:12.148 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:25:12.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:25:12.188 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:12.184+0000 7f6728d4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:12.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:12.220+0000 7f6728d4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:12.224 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:12.220+0000 7f6728d4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:12.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-04T19:25:12.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-04T19:25:13.473 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:13.468+0000 7f6728d4b7c0 -1 Falling back to public interface 2024-09-04T19:25:16.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:16.560+0000 7f6728d4b7c0 -1 osd.1 29 log_to_monitors true 2024-09-04T19:25:17.592 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery divergent 2024-09-04T19:25:17.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-04T19:25:17.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:18.002 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:18.002 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:18.003 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:18.003 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:18.003 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:18.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:25:18.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:25:18.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok set_recovery_delay 100000 2024-09-04T19:25:18.136 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:25:18.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-04T19:25:18.825 INFO:tasks.workunit.client.0.smithi174.stderr:noup is unset 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:25:18.846 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:25:18.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:25:18.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:25:18.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:25:19.390 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1313135797,v1:127.0.0.1:6803/1313135797] [v2:127.0.0.1:6804/1313135797,v1:127.0.0.1:6805/1313135797] exists,up 5057219b-c82e-4a4d-9b96-c5a119c90e41 2024-09-04T19:25:19.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:25:19.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:25:19.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:25:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-04T19:25:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:25:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:25:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:25:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:25:19.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:25:19.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:25:19.395 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:25:19.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:25:19.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:25:19.855 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2888785372,v1:127.0.0.1:6819/2888785372] [v2:127.0.0.1:6820/2888785372,v1:127.0.0.1:6821/2888785372] exists,up b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:25:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:25:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:25:19.858 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:25:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:25:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:25:20.336 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/154937205,v1:127.0.0.1:6811/154937205] [v2:127.0.0.1:6812/154937205,v1:127.0.0.1:6813/154937205] exists,up 22cd6806-54c3-4f4b-aa24-f1086b7ea47e 2024-09-04T19:25:20.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:25:20.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:25:20.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:25:20.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:20.746 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:20.746 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:20.746 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:20.746 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:20.746 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:20.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-04T19:25:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:wait for peering 2024-09-04T19:25:20.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:21.162 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:21.162 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:21.162 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:21.162 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:21.162 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:21.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-04T19:25:21.319 INFO:tasks.workunit.client.0.smithi174.stdout:killing divergent 1 2024-09-04T19:25:21.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-04T19:25:21.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:21.725 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:21.725 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-04T19:23:55.370778+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-06T01:55:16.758100+0000 0 0 2024-09-04T19:25:21.725 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:21.725 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:21.726 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:21.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-04T19:25:21.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:25:21.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:25:21.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:25:21.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:25:21.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:25:21.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27364.out 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27364.out 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-04T19:25:21.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27364.out 2024-09-04T19:25:23.105 INFO:tasks.workunit.client.0.smithi174.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-04T19:25:23.107 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-04T19:25:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-04T19:25:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-04T19:25:23.109 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-04T19:25:23.109 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-04T19:25:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-04T19:25:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-04T19:25:23.111 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:169157fa:::existing_81:head# 2024-09-04T19:25:23.111 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:17731382:::existing_58:head# 2024-09-04T19:25:23.112 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-04T19:25:23.112 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-04T19:25:23.113 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-04T19:25:23.113 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-04T19:25:23.114 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-04T19:25:23.114 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:28566edb:::existing_78:head# 2024-09-04T19:25:23.115 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:2a101182:::existing_20:head# 2024-09-04T19:25:23.115 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-04T19:25:23.115 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-04T19:25:23.116 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-04T19:25:23.116 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-04T19:25:23.117 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-04T19:25:23.117 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-04T19:25:23.118 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-04T19:25:23.118 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-04T19:25:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-04T19:25:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-04T19:25:23.120 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:53588912:::existing_9:head# 2024-09-04T19:25:23.120 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-04T19:25:23.120 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-04T19:25:23.121 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-04T19:25:23.121 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-04T19:25:23.121 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:602f83fe:::foo:head# 2024-09-04T19:25:23.122 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-04T19:25:23.122 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:628918ca:::existing_79:head# 2024-09-04T19:25:23.123 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-04T19:25:23.123 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-04T19:25:23.124 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-04T19:25:23.124 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-04T19:25:23.125 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-04T19:25:23.125 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-04T19:25:23.126 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-04T19:25:23.126 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-04T19:25:23.126 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:750ace21:::existing_19:head# 2024-09-04T19:25:23.127 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-04T19:25:23.127 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:79881152:::existing_63:head# 2024-09-04T19:25:23.128 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-04T19:25:23.128 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:81490677:::existing_76:head# 2024-09-04T19:25:23.128 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-04T19:25:23.129 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-04T19:25:23.129 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-04T19:25:23.130 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-04T19:25:23.130 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:92731e71:::existing_89:head# 2024-09-04T19:25:23.130 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:928205cf:::existing_82:head# 2024-09-04T19:25:23.131 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:93be2267:::existing_24:head# 2024-09-04T19:25:23.132 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-04T19:25:23.132 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-04T19:25:23.132 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-04T19:25:23.133 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-04T19:25:23.133 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-04T19:25:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a504a533:::existing_90:head# 2024-09-04T19:25:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a691337e:::existing_61:head# 2024-09-04T19:25:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-04T19:25:23.135 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-04T19:25:23.135 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-04T19:25:23.136 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-04T19:25:23.137 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-04T19:25:23.137 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:b343e277:::existing_88:head# 2024-09-04T19:25:23.137 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-04T19:25:23.137 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-04T19:25:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-04T19:25:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-04T19:25:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:bebec877:::existing_74:head# 2024-09-04T19:25:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-04T19:25:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:c156033e:::existing_100:head# 2024-09-04T19:25:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-04T19:25:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-04T19:25:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-04T19:25:23.142 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-04T19:25:23.142 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-04T19:25:23.142 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-04T19:25:23.143 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-04T19:25:23.143 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-04T19:25:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-04T19:25:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-04T19:25:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-04T19:25:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-04T19:25:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-04T19:25:23.146 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-04T19:25:23.146 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:e255b710:::existing_55:head# 2024-09-04T19:25:23.147 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-04T19:25:23.147 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-04T19:25:23.147 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-04T19:25:23.148 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:f6809456:::existing_11:head# 2024-09-04T19:25:23.149 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-04T19:25:23.149 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-04T19:25:23.149 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-04T19:25:23.150 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-04T19:25:23.150 INFO:tasks.workunit.client.0.smithi174.stderr:Export successful 2024-09-04T19:25:23.151 INFO:tasks.workunit.client.0.smithi174.stdout: marking collection for removal 2024-09-04T19:25:23.151 INFO:tasks.workunit.client.0.smithi174.stdout:setting '_remove' omap key 2024-09-04T19:25:23.151 INFO:tasks.workunit.client.0.smithi174.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-04T19:25:23.305 INFO:tasks.workunit.client.0.smithi174.stderr:Remove successful 2024-09-04T19:25:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27364.out 2024-09-04T19:25:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:25:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:25:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-04T19:25:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:25:23.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-04T19:25:23.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27364.out 2024-09-04T19:25:24.870 INFO:tasks.workunit.client.0.smithi174.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-04T19:25:24.871 INFO:tasks.workunit.client.0.smithi174.stdout:Importing pgid 1.0 2024-09-04T19:25:24.871 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-04T19:25:24.871 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:24.934 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-04T19:25:24.934 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:24.967 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-04T19:25:24.967 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.000 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-04T19:25:25.001 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.034 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-04T19:25:25.034 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.067 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-04T19:25:25.068 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.101 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-04T19:25:25.101 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.135 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:169157fa:::existing_81:head# 2024-09-04T19:25:25.135 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.168 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:17731382:::existing_58:head# 2024-09-04T19:25:25.168 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.213 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-04T19:25:25.213 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.243 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-04T19:25:25.244 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.277 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-04T19:25:25.277 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.310 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-04T19:25:25.310 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.344 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-04T19:25:25.344 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.377 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:28566edb:::existing_78:head# 2024-09-04T19:25:25.377 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.410 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:2a101182:::existing_20:head# 2024-09-04T19:25:25.411 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.444 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-04T19:25:25.444 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.477 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-04T19:25:25.478 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.511 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-04T19:25:25.511 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.544 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-04T19:25:25.544 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.578 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-04T19:25:25.578 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.611 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-04T19:25:25.611 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.644 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-04T19:25:25.645 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.678 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-04T19:25:25.678 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.711 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-04T19:25:25.711 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.745 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-04T19:25:25.745 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.787 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:53588912:::existing_9:head# 2024-09-04T19:25:25.787 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.845 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-04T19:25:25.845 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.887 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-04T19:25:25.887 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.912 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-04T19:25:25.912 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.937 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-04T19:25:25.937 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.962 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:602f83fe:::foo:head# 2024-09-04T19:25:25.962 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:25.987 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-04T19:25:25.988 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.013 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:628918ca:::existing_79:head# 2024-09-04T19:25:26.013 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.038 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-04T19:25:26.038 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.063 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-04T19:25:26.063 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.088 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-04T19:25:26.088 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.113 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-04T19:25:26.113 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.138 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-04T19:25:26.138 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.163 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-04T19:25:26.163 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.189 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-04T19:25:26.189 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.214 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-04T19:25:26.214 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.239 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:750ace21:::existing_19:head# 2024-09-04T19:25:26.239 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.264 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-04T19:25:26.264 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.289 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:79881152:::existing_63:head# 2024-09-04T19:25:26.289 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.339 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-04T19:25:26.339 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.398 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:81490677:::existing_76:head# 2024-09-04T19:25:26.398 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.440 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-04T19:25:26.441 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.465 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-04T19:25:26.465 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.490 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-04T19:25:26.490 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.515 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-04T19:25:26.515 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.540 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:92731e71:::existing_89:head# 2024-09-04T19:25:26.541 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.574 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:928205cf:::existing_82:head# 2024-09-04T19:25:26.574 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.607 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:93be2267:::existing_24:head# 2024-09-04T19:25:26.607 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.641 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-04T19:25:26.641 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.674 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-04T19:25:26.674 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.708 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-04T19:25:26.708 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.741 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-04T19:25:26.741 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.775 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-04T19:25:26.775 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.808 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a504a533:::existing_90:head# 2024-09-04T19:25:26.808 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.841 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a691337e:::existing_61:head# 2024-09-04T19:25:26.842 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.875 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-04T19:25:26.875 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.908 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-04T19:25:26.908 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.942 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-04T19:25:26.942 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:26.975 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-04T19:25:26.975 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.034 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-04T19:25:27.034 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.067 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:b343e277:::existing_88:head# 2024-09-04T19:25:27.067 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.101 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-04T19:25:27.101 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.134 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-04T19:25:27.134 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.168 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-04T19:25:27.168 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.210 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-04T19:25:27.210 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.243 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:bebec877:::existing_74:head# 2024-09-04T19:25:27.243 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.277 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-04T19:25:27.277 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.310 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:c156033e:::existing_100:head# 2024-09-04T19:25:27.310 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.343 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-04T19:25:27.343 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.377 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-04T19:25:27.377 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.410 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-04T19:25:27.410 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.452 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-04T19:25:27.452 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.494 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-04T19:25:27.494 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.527 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-04T19:25:27.528 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.561 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-04T19:25:27.561 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.594 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-04T19:25:27.594 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.628 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-04T19:25:27.628 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.661 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-04T19:25:27.661 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.695 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-04T19:25:27.695 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.728 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-04T19:25:27.728 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.761 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-04T19:25:27.762 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.828 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-04T19:25:27.829 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.881 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:e255b710:::existing_55:head# 2024-09-04T19:25:27.881 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.912 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-04T19:25:27.912 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.945 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-04T19:25:27.946 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:27.979 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-04T19:25:27.979 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.029 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:f6809456:::existing_11:head# 2024-09-04T19:25:28.029 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.062 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-04T19:25:28.063 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.088 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-04T19:25:28.088 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.113 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-04T19:25:28.113 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.138 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-04T19:25:28.138 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:25:28.163 INFO:tasks.workunit.client.0.smithi174.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-04T19:25:28.188 INFO:tasks.workunit.client.0.smithi174.stdout:Import successful 2024-09-04T19:25:28.726 INFO:tasks.workunit.client.0.smithi174.stdout:reviving divergent 1 2024-09-04T19:25:28.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-04T19:25:28.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:29.129 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:29.129 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-09-04T19:25:20.168922+0000 47'102 47:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:25:29.129 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:29.129 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:29.130 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:25:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:29.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:25:29.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:25:29.151 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:25:29.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:25:29.189 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:29.184+0000 7fc3fab3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:29.191 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:29.188+0000 7fc3fab3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:29.192 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:29.188+0000 7fc3fab3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:25:29.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:25:29.636 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:25:29.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:25:29.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:25:30.069 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2888785372,v1:127.0.0.1:6819/2888785372] [v2:127.0.0.1:6820/2888785372,v1:127.0.0.1:6821/2888785372] exists,up b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:25:30.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:25:30.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:25:30.071 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:25:30.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:25:30.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:25:30.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:25:30.412 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:30.408+0000 7fc3fab3d7c0 -1 Falling back to public interface 2024-09-04T19:25:30.522 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2888785372,v1:127.0.0.1:6819/2888785372] [v2:127.0.0.1:6820/2888785372,v1:127.0.0.1:6821/2888785372] exists,up b538fa63-4022-4fdf-8310-4533b70a9c58 2024-09-04T19:25:30.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:25:30.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:25:30.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:25:30.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-04T19:25:33.286 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:25:33.280+0000 7fc3fab3d7c0 -1 osd.1 47 log_to_monitors true 2024-09-04T19:25:50.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-04T19:25:50.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:25:50.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:25:50.526 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:50.527 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:50.527 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:50.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:25:50.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-04T19:25:50.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T19:25:50.643 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:25:50.643 INFO:tasks.workunit.client.0.smithi174.stdout: "ops": [], 2024-09-04T19:25:50.643 INFO:tasks.workunit.client.0.smithi174.stdout: "num_ops": 0 2024-09-04T19:25:50.643 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:25:50.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-04T19:25:50.655 INFO:tasks.workunit.client.0.smithi174.stdout:allowing recovery 2024-09-04T19:25:50.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:51.049 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:51.049 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-04T19:25:34.807688+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T21:42:04.543312+0000 0 0 2024-09-04T19:25:51.049 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:51.049 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:51.049 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:51.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-04T19:25:51.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-04T19:25:51.222 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:25:51.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-04T19:25:51.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-04T19:25:51.400 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:25:51.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-04T19:25:51.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-04T19:25:51.566 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:25:51.580 INFO:tasks.workunit.client.0.smithi174.stdout:reading divergent objects 2024-09-04T19:25:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-04T19:25:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-04T19:25:51.970 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:25:51.970 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-04T19:25:34.807688+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:22:55.400242+0000 0'0 2024-09-04T19:22:55.400242+0000 0 0 periodic scrub scheduled @ 2024-09-05T21:42:04.543312+0000 0 0 2024-09-04T19:25:51.970 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:25:51.970 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:25:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:25:51.986 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-04T19:25:51.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-04T19:25:51.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:25:51.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-04T19:25:52.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:25:52.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-04T19:25:52.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:25:52.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-04T19:25:52.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:25:52.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-04T19:25:52.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-04T19:25:52.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:25:52.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-04T19:25:52.169 INFO:tasks.workunit.client.0.smithi174.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-04T19:25:52.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:25:52.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-04T19:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-04T19:25:52.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-04T19:25:52.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-04T19:25:52.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2024-09-04T19:23:29.394270+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2024-09-04T19:23:29.394270+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4752.0:1 2024-09-04T19:24:32.843877+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4621.0:1 2024-09-04T19:23:29.368785+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4621.0:1 2024-09-04T19:23:29.368785+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-04T19:25:52.212 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-09-04T19:23:29.389207+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-09-04T19:23:29.389207+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4632.0:1 2024-09-04T19:23:29.389207+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-09-04T19:23:29.372625+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.213 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-09-04T19:23:29.372625+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2024-09-04T19:23:29.372625+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-04T19:23:29.388566+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.214 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-04T19:23:29.388566+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-04T19:23:29.388566+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2024-09-04T19:23:29.385077+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2024-09-04T19:23:29.385077+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:25:52.215 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:25:52.216 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-04T19:25:52.216 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-04T19:25:52.216 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:25:20.084+0000 7f670ccd5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-04T19:25:52.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-04T19:25:52.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:25:52.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-04T19:25:52.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:25:52.224 INFO:tasks.workunit.client.0.smithi174.stdout:success 2024-09-04T19:25:52.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-04T19:25:52.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27364.out 2024-09-04T19:25:52.225 INFO:tasks.workunit.client.0.smithi174.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27364.out': No such file or directory 2024-09-04T19:25:52.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-04T19:25:52.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-04T19:25:52.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-04T19:25:52.735 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' does not exist 2024-09-04T19:25:52.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-04T19:25:52.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:25:52.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:25:52.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:25:52.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:25:52.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:25:58.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:25:58.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:25:58.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:25:58.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:25:58.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:25:58.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:25:58.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:25:58.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:25:58.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:25:58.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:25:58.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:25:58.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:25:58.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:25:58.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:25:58.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:25:58.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:25:58.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:25:58.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:25:58.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:25:58.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:25:58.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:25:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:25:58.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:25:58.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:25:58.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:25:58.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:25:58.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:25:58.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:25:58.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:25:58.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:25:58.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:25:58.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:25:58.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:25:58.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:25:58.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:25:58.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:25:58.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:25:58.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:25:58.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:25:58.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:25:58.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:25:58.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:25:58.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-04T19:25:58.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:25:58.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27364 2024-09-04T19:25:58.129 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-04T19:25:58.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-04T19:25:58.131 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-04T19:25:58.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:25:58.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:25:58.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:25:58.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-04T19:25:58.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-04T19:25:58.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:25:58.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:25:58.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:25:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:25:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:58.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:25:58.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:25:58.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:25:58.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:25:58.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:25:58.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:25:58.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.733 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:25:58.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:25:58.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get fsid 2024-09-04T19:25:58.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:25:58.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:25:58.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:25:58.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:25:58.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:25:58.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get mon_host 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:25:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-04T19:25:58.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:25:59.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:59.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:25:59.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:25:59.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:25:59.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:25:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:25:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:25:59.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:25:59.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:25:59.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e58308e4-8e6d-41b5-8d4b-0f1787630e30' 2024-09-04T19:25:59.935 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:25:59.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:25:59.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHtNhm/vQNORAAF+0OwyT7/gfr2cKG+V7BlQ== 2024-09-04T19:25:59.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHtNhm/vQNORAAF+0OwyT7/gfr2cKG+V7BlQ=="}' 2024-09-04T19:25:59.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e58308e4-8e6d-41b5-8d4b-0f1787630e30 -i td/divergent-priors/0/new.json 2024-09-04T19:26:00.258 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:26:00.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-04T19:26:00.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHtNhm/vQNORAAF+0OwyT7/gfr2cKG+V7BlQ== --osd-uuid e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:26:00.312 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:00.308+0000 7f59ff3e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:00.326 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:00.320+0000 7f59ff3e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:00.327 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:00.324+0000 7f59ff3e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:00.327 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:00.324+0000 7f59ff3e77c0 -1 bdev(0x5565cd40ea80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:26:00.327 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:00.324+0000 7f59ff3e77c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-04T19:26:06.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-04T19:26:06.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:26:06.310 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:26:06.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:26:06.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:26:06.888 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:26:06.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:26:06.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:26:06.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:26:06.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:26:06.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:26:06.926 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:06.920+0000 7fb16b1b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:06.930 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:06.924+0000 7fb16b1b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:06.931 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:06.928+0000 7fb16b1b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:07.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:26:07.341 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:26:07.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:07.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:07.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:07.989 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:07.984+0000 7fb16b1b77c0 -1 Falling back to public interface 2024-09-04T19:26:08.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:08.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:08.812 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:26:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:26:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:08.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:09.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:10.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:10.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:10.246 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:26:10.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:26:10.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:10.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:10.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:10.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:10.724+0000 7fb16b1b77c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:26:11.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:11.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:11.663 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:26:11.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:26:11.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:11.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:12.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:13.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:13.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:13.100 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:26:13.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:26:13.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:13.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:13.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:14.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:14.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:14.598 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:26:14.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:26:14.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:14.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:15.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:16.044 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:26:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:26:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:16.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:17.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:17.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:17.609 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:26:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:26:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:18.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:19.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:19.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:19.048 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:26:19.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:26:19.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:19.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:19.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:20.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:20.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:20.597 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:26:20.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:26:20.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:21.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:22.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:22.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:22.081 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:26:22.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:26:22.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:22.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:22.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:23.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:23.721 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:26:23.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:23.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:26:23.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:23.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:24.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:25.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:25.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:25.148 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:26:25.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:26:25.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:25.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:25.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:26.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:26.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:26.813 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:26:26.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:26:26.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:26.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:27.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:28.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:28.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:28.865 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:26:28.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:26:28.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:28.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:29.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:30.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:30.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:30.824 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:26:30.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:26:30.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:30.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:31.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:32.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:32.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:32.891 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:26:32.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:26:32.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:32.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:26:33.387 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1732262552,v1:127.0.0.1:6803/1732262552] [v2:127.0.0.1:6804/1732262552,v1:127.0.0.1:6805/1732262552] exists,up e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:26:33.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:26:33.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:26:33.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:26:33.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:26:33.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:26:33.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:26:33.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:26:33.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:26:33.394 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:26:33.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2e03be93-3167-48bf-8313-79cddea604a4' 2024-09-04T19:26:33.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:26:33.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBptNhmbQwJGRAAdVbm7zcocGRvVBF1A5JzWQ== 2024-09-04T19:26:33.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBptNhmbQwJGRAAdVbm7zcocGRvVBF1A5JzWQ=="}' 2024-09-04T19:26:33.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e03be93-3167-48bf-8313-79cddea604a4 -i td/divergent-priors/1/new.json 2024-09-04T19:26:34.041 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:26:34.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-04T19:26:34.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBptNhmbQwJGRAAdVbm7zcocGRvVBF1A5JzWQ== --osd-uuid 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:26:34.102 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:34.099+0000 7fe4eae9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:34.121 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:34.115+0000 7fe4eae9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:34.122 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:34.119+0000 7fe4eae9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:34.123 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:34.119+0000 7fe4eae9b7c0 -1 bdev(0x5573532f8a80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:26:34.123 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:34.119+0000 7fe4eae9b7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-04T19:26:40.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-04T19:26:40.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:26:40.070 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:26:40.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:26:40.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:26:40.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:26:40.635 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:26:40.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:26:40.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:26:40.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:26:40.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:26:40.675 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:40.671+0000 7fc13495a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:40.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:40.671+0000 7fc13495a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:40.678 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:40.675+0000 7fc13495a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:26:41.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:26:41.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:26:41.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:26:41.086 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:26:41.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:26:41.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:26:41.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:41.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:26:41.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:41.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:41.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:41.669 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:41.663+0000 7fc13495a7c0 -1 Falling back to public interface 2024-09-04T19:26:42.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:42.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:42.518 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:26:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:26:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:42.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:43.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:43.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:43.957 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:26:43.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:26:43.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:43.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:44.245 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:26:44.239+0000 7fc13495a7c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:26:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:45.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:45.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:45.418 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:26:45.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:26:45.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:45.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:45.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:46.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:46.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:46.985 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:26:46.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:26:46.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:46.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:47.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:48.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:48.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:48.436 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:26:48.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:26:48.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:48.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:49.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:50.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:50.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:50.025 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:26:50.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:26:50.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:50.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:50.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:51.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:51.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:51.459 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:26:51.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:26:51.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:51.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:52.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:53.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:53.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:53.039 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:26:53.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:26:53.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:53.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:53.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:54.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:54.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:54.474 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:26:54.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:26:54.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:54.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:55.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:56.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:56.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:56.030 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:26:56.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:26:56.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:56.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:56.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:57.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:57.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:57.570 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:26:57.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:26:57.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:57.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:26:58.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:26:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:26:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:26:59.384 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:26:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:26:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:26:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:27:00.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:01.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:01.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:01.163 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:27:01.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:27:01.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:01.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:27:02.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:03.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:03.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:03.180 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:27:03.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:27:03.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:03.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:27:03.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:04.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:04.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:04.961 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:27:04.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:27:04.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:04.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:27:05.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:06.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:06.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:06.441 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:27:06.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:27:06.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:06.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:27:07.008 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/978199495,v1:127.0.0.1:6811/978199495] [v2:127.0.0.1:6812/978199495,v1:127.0.0.1:6813/978199495] exists,up 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:27:07.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:27:07.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:27:07.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:27:07.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:27:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:27:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bb73362d-2836-4717-a60d-80254994227d' 2024-09-04T19:27:07.015 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:27:07.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:27:07.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLtNhmoJuEAhAA4aFT5q7NWaTO5sxb38IkvQ== 2024-09-04T19:27:07.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLtNhmoJuEAhAA4aFT5q7NWaTO5sxb38IkvQ=="}' 2024-09-04T19:27:07.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb73362d-2836-4717-a60d-80254994227d -i td/divergent-priors/2/new.json 2024-09-04T19:27:07.567 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:27:07.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-04T19:27:07.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLtNhmoJuEAhAA4aFT5q7NWaTO5sxb38IkvQ== --osd-uuid bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:27:07.629 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:07.623+0000 7f1045e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:07.632 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:07.627+0000 7f1045e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:07.633 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:07.627+0000 7f1045e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:07.633 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:07.627+0000 7f1045e2f7c0 -1 bdev(0x5589789baa80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:27:07.633 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:07.627+0000 7f1045e2f7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-04T19:27:13.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-04T19:27:13.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:27:13.890 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:27:13.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:27:13.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:27:14.451 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:27:14.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:27:14.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:27:14.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:27:14.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:27:14.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:27:14.492 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:14.491+0000 7f0397ca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:14.516 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:14.515+0000 7f0397ca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:14.516 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:14.515+0000 7f0397ca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:27:14.971 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:27:14.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:14.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:15.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:15.581 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:15.579+0000 7f0397ca17c0 -1 Falling back to public interface 2024-09-04T19:27:16.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:16.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:16.389 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:27:16.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:27:16.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:16.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:16.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:17.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:17.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:17.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:27:17.849 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:27:17.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:17.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:18.244 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:27:18.243+0000 7f0397ca17c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:27:18.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:19.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:19.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:19.285 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:27:19.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:27:19.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:19.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:19.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:20.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:20.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:20.739 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:27:20.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:27:20.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:20.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:21.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:27:22.220 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:27:22.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:22.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:22.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:27:23.674 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:27:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:23.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:24.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:25.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:25.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:25.234 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:27:25.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:27:25.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:25.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:25.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:26.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:26.755 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:27:26.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:26.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:27:26.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:26.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:27.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:28.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:28.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:28.226 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:27:28.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:27:28.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:28.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:28.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:29.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:29.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:29.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:27:29.664 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:27:29.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:29.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:30.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:31.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:31.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:31.095 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:27:31.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:27:31.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:31.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:31.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:32.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:32.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:32.786 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:27:32.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:27:32.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:32.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:34.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:35.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:35.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:35.016 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:27:35.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:27:35.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:35.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:35.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:36.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:36.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:36.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:27:36.752 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:27:36.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:36.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:37.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:38.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:38.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:38.781 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:27:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:27:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:39.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:27:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:27:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:27:40.845 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:27:40.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:27:40.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:27:40.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:27:41.368 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3332327959,v1:127.0.0.1:6819/3332327959] [v2:127.0.0.1:6820/3332327959,v1:127.0.0.1:6821/3332327959] exists,up bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:27:41.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:27:41.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:27:41.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:27:41.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-04T19:27:41.991 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-04T19:27:42.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-04T19:27:42.795 INFO:tasks.workunit.client.0.smithi174.stderr:noin is set 2024-09-04T19:27:42.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-04T19:27:43.489 INFO:tasks.workunit.client.0.smithi174.stderr:nodown is set 2024-09-04T19:27:43.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-04T19:27:43.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-04T19:27:44.042 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' already exists 2024-09-04T19:27:44.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:27:45.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-04T19:27:45.744 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 size to 3 2024-09-04T19:27:45.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-04T19:27:46.489 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 min_size to 2 2024-09-04T19:27:46.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-04T19:27:47.201 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 pg_autoscale_mode to on 2024-09-04T19:27:47.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-04T19:27:47.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1725478067 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:27:47.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:47.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:27:47.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-09-04T19:27:47.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-09-04T19:27:47.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-09-04T19:27:47.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:47.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:27:48.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-09-04T19:27:48.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-09-04T19:27:48.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-09-04T19:27:48.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:48.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:27:48.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542151 2024-09-04T19:27:48.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542151 2024-09-04T19:27:48.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971 2-60129542151' 2024-09-04T19:27:48.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:48.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-09-04T19:27:48.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:48.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:27:48.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-09-04T19:27:48.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:48.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-09-04T19:27:48.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-04T19:27:48.170 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836499 2024-09-04T19:27:48.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:27:48.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-04T19:27:48.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:27:49.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:27:49.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:27:50.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-09-04T19:27:50.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-04T19:27:50.081 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672971 2024-09-04T19:27:50.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:27:50.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-04T19:27:50.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:50.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542151 2024-09-04T19:27:50.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:50.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:27:50.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542151 2024-09-04T19:27:50.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:50.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542151 2024-09-04T19:27:50.507 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 60129542151 2024-09-04T19:27:50.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2024-09-04T19:27:50.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:27:50.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542151 -lt 60129542151 2024-09-04T19:27:50.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-04T19:27:50.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:27:50.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:27:50.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:27:50.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:27:50.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:27:50.982 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:27:50.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:27:50.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:27:50.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:27:51.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:27:51.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:27:51.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:27:51.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:27:51.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:27:51.091 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:27:51.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:27:51.525 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:27:51.525 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:27:51.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:27:51.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:51.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:27:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-09-04T19:27:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-09-04T19:27:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-09-04T19:27:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:51.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:27:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-09-04T19:27:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-09-04T19:27:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-09-04T19:27:51.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:27:51.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:27:52.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542154 2024-09-04T19:27:52.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542154 2024-09-04T19:27:52.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-60129542154' 2024-09-04T19:27:52.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:52.022 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-09-04T19:27:52.022 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:52.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:27:52.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-09-04T19:27:52.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:52.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-09-04T19:27:52.024 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836502 2024-09-04T19:27:52.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-04T19:27:52.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:27:52.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-04T19:27:52.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:27:53.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:27:53.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:27:53.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-04T19:27:53.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:53.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-09-04T19:27:53.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:53.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:27:53.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-09-04T19:27:53.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:53.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-09-04T19:27:53.866 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672973 2024-09-04T19:27:53.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-04T19:27:53.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:27:54.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-04T19:27:54.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:27:54.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542154 2024-09-04T19:27:54.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:27:54.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:27:54.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542154 2024-09-04T19:27:54.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:27:54.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542154 2024-09-04T19:27:54.305 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 60129542154 2024-09-04T19:27:54.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2024-09-04T19:27:54.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:27:54.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542154 -lt 60129542154 2024-09-04T19:27:54.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:27:54.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:27:54.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:27:55.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:27:55.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:27:55.367 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:27:55.367 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:27:55.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:27:55.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:27:55.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:27:55.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:27:55.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:27:55.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:27:55.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:27:55.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:27:56.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:27:56.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:27:56.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:27:56.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-04T19:27:56.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-04T19:27:56.693 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:27:56.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-04T19:27:56.709 INFO:tasks.workunit.client.0.smithi174.stdout:primary and soon to be divergent is 1 2024-09-04T19:27:56.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-04T19:27:56.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:27:57.094 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:27:57.095 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-04T19:27:45.343858+0000 0'0 29:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-06T05:35:47.681738+0000 0 0 2024-09-04T19:27:57.095 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:27:57.095 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:27:57.095 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:27:57.111 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-04T19:27:57.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1725478077 2024-09-04T19:27:57.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1725478077 - 1725478067 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-04T19:27:57.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-04T19:27:57.116 INFO:tasks.workunit.client.0.smithi174.stdout:writing initial objects 2024-09-04T19:27:57.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-04T19:27:57.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-04T19:27:57.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-04T19:27:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-04T19:27:57.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-04T19:27:57.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-04T19:27:57.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-04T19:27:57.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:57.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-04T19:27:58.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:58.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-04T19:27:58.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:58.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-04T19:27:58.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:58.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-04T19:27:58.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:58.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-04T19:27:58.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:58.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-04T19:27:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-04T19:27:59.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-04T19:27:59.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-04T19:27:59.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-04T19:27:59.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-04T19:27:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-04T19:27:59.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-04T19:27:59.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:27:59.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-04T19:28:00.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-04T19:28:00.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-04T19:28:00.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-04T19:28:00.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-04T19:28:00.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-04T19:28:00.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-04T19:28:00.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:00.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-04T19:28:01.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-04T19:28:01.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-04T19:28:01.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-04T19:28:01.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-04T19:28:01.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-04T19:28:01.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:01.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-04T19:28:02.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-04T19:28:02.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-04T19:28:02.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-04T19:28:02.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-04T19:28:02.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-04T19:28:02.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-04T19:28:02.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:02.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-04T19:28:03.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-04T19:28:03.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-04T19:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-04T19:28:03.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-04T19:28:03.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-04T19:28:03.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:03.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-04T19:28:04.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:04.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-04T19:28:04.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:04.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-04T19:28:04.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:04.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-04T19:28:04.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:04.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-04T19:28:04.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:04.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-04T19:28:05.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-04T19:28:05.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-04T19:28:05.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-04T19:28:05.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-04T19:28:05.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-04T19:28:05.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-04T19:28:05.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:05.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-04T19:28:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-04T19:28:06.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:06.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-04T19:28:06.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:06.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-04T19:28:06.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:06.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-04T19:28:06.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:06.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-04T19:28:07.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:07.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-04T19:28:08.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-04T19:28:08.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-04T19:28:08.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-04T19:28:08.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-04T19:28:08.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-04T19:28:08.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:08.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-04T19:28:09.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:09.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-04T19:28:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-04T19:28:09.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:09.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-04T19:28:09.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:09.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-04T19:28:09.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:09.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-04T19:28:10.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-04T19:28:10.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-04T19:28:10.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-04T19:28:10.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-04T19:28:10.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-04T19:28:10.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:10.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-04T19:28:11.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-04T19:28:11.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-04T19:28:11.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-04T19:28:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-04T19:28:11.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-04T19:28:11.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:11.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-04T19:28:12.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-04T19:28:12.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-04T19:28:12.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-04T19:28:12.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-04T19:28:12.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-04T19:28:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-04T19:28:13.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-04T19:28:13.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-04T19:28:13.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-04T19:28:13.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-04T19:28:13.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-04T19:28:13.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:13.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-04T19:28:14.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-04T19:28:14.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-04T19:28:14.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:28:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-04T19:28:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:28:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:28:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:28:14.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:28:14.220 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:28:14.220 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:28:14.220 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:28:14.220 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:28:14.220 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:28:14.296 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:28:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:28:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:28:14.739 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:28:14.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:28:14.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:14.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:28:14.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-09-04T19:28:14.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-09-04T19:28:14.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-09-04T19:28:14.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:14.914 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:28:15.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-09-04T19:28:15.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-09-04T19:28:15.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672980' 2024-09-04T19:28:15.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:15.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:28:15.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542160 2024-09-04T19:28:15.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542160 2024-09-04T19:28:15.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672980 2-60129542160' 2024-09-04T19:28:15.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:28:15.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-09-04T19:28:15.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:28:15.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:28:15.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-09-04T19:28:15.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:28:15.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-09-04T19:28:15.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-04T19:28:15.258 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836509 2024-09-04T19:28:15.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:28:15.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-04T19:28:15.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:28:16.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:28:16.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:28:17.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-04T19:28:17.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:28:17.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-09-04T19:28:17.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:28:17.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:28:17.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-09-04T19:28:17.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:28:17.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-09-04T19:28:17.173 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672980 2024-09-04T19:28:17.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-04T19:28:17.173 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:28:17.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-04T19:28:17.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:28:17.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542160 2024-09-04T19:28:17.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:28:17.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:28:17.654 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542160 2024-09-04T19:28:17.654 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:28:17.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542160 2024-09-04T19:28:17.656 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 60129542160 2024-09-04T19:28:17.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-09-04T19:28:17.656 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:28:18.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542160 -lt 60129542160 2024-09-04T19:28:18.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:28:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:18.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-04T19:28:18.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:18.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-04T19:28:19.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:19.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:19.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:19.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-04T19:28:19.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-04T19:28:20.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-04T19:28:20.335 INFO:tasks.workunit.client.0.smithi174.stdout:blackholing osds 0 2 2024-09-04T19:28:20.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-04T19:28:20.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:28:20.760 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:28:20.760 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 2 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:28:15.729002+0000 32'63 34:111 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.760 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:07.791752+0000 32'64 34:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.760 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:15.728562+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.760 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:28:15.728558+0000 32'63 34:111 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:28:07.788701+0000 30'62 34:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:15.728045+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:10.687604+0000 32'64 34:115 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:56:09.814972+0000 1 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:28:15.727978+0000 30'62 34:99 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:07.784879+0000 32'64 34:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:15.641088+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-09-04T19:28:11.700144+0000 30'62 34:113 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:19:37.944653+0000 2 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-09-04T19:28:15.640314+0000 32'63 34:119 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:19:49.464566+0000 4 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-09-04T19:28:14.652047+0000 32'63 34:114 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:14.651734+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:52:27.579821+0000 1 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:28:07.788292+0000 32'63 34:106 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.761 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:15.638454+0000 32'64 34:111 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:13.619474+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:28:07.786902+0000 30'62 34:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:09.682416+0000 32'63 34:114 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:09:05.620365+0000 2 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:15.642126+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:07.788853+0000 32'64 34:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:28:15.640292+0000 32'63 34:111 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:28:07.790881+0000 32'63 34:106 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-09-04T19:28:15.945931+0000 30'62 34:107 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:07:11.305834+0000 3 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:28:07.786163+0000 30'62 34:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:15.727574+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:15.727617+0000 32'64 34:115 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:00:32.441813+0000 3 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:28:15.728173+0000 32'63 34:111 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.762 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:28:17.629541+0000 32'64 34:115 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:27:44.592861+0000 4 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:28:15.641560+0000 30'62 34:99 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:07.790184+0000 32'64 34:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:28:15.725813+0000 30'62 34:99 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:28:16.657385+0000 32'63 34:114 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:50:50.132642+0000 4 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:15.728595+0000 32'64 34:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:28:20.763 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:28:20.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:20.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-04T19:28:20.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:28:20.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:28:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:28:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:28:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:28:20.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:28:20.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:28:20.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-04T19:28:20.886 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:28:20.886 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:28:20.886 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:28:20.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:20.899 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-04T19:28:20.899 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:28:20.899 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:28:20.900 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:28:20.900 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:28:20.900 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:28:20.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:28:20.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:28:20.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-04T19:28:21.005 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:28:21.005 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:28:21.005 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:28:21.018 INFO:tasks.workunit.client.0.smithi174.stdout:create new divergent objects 2024-09-04T19:28:21.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-04T19:28:21.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-04T19:28:21.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:28:21.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:28:21.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-04T19:28:21.020 INFO:tasks.workunit.client.0.smithi174.stdout:writing divergent objects 2024-09-04T19:28:21.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-04T19:28:21.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-04T19:28:21.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-04T19:28:21.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-04T19:28:21.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-04T19:28:21.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-04T19:28:21.022 INFO:tasks.workunit.client.0.smithi174.stdout:remove divergent objects 2024-09-04T19:28:21.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-04T19:28:21.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-04T19:28:21.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-04T19:28:21.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-04T19:28:21.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-04T19:28:21.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-04T19:28:21.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-04T19:28:21.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-04T19:28:21.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-04T19:28:21.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-04T19:28:21.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-04T19:28:21.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-04T19:28:31.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-04T19:28:31.036 INFO:tasks.workunit.client.0.smithi174.stderr:rados: no process found 2024-09-04T19:28:31.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-04T19:28:31.036 INFO:tasks.workunit.client.0.smithi174.stdout:killing all the osds 2024-09-04T19:28:31.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:28:31.525 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:28:31.525 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:24.826029+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:24.825823+0000 32'63 2024-09-04T19:28:24.825823+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:14:56.467872+0000 3 0 2024-09-04T19:28:31.525 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:27.265412+0000 32'64 42:117 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:22.707656+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:31.525 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:28:19.796792+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:32:31.482011+0000 4 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:25.794128+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:25.793935+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:08:42.989229+0000 3 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 active+clean 2024-09-04T19:28:24.795995+0000 30'62 42:123 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:24.795794+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:04:35.201731+0000 3 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 5 0 0 0 0 3435 0 0 17 48 17 active+clean 2024-09-04T19:28:24.868232+0000 37'65 40:137 [2,0,1] 2 [2,0,1] 2 37'65 2024-09-04T19:28:24.868034+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:12:52.349731+0000 5 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:10.687604+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:54:21.986691+0000 1 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-09-04T19:28:26.855832+0000 30'62 42:129 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:26.855615+0000 30'62 2024-09-04T19:28:26.855615+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:35:21.651481+0000 2 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:28:27.265000+0000 32'64 42:117 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:26.794555+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-09-04T19:28:18.874124+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:27:06.152174+0000 6 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-09-04T19:28:11.700144+0000 30'62 42:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:42:58.767703+0000 2 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-09-04T19:28:21.611014+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:07:07.593683+0000 4 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-09-04T19:28:19.797062+0000 32'63 42:135 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:40:44.905826+0000 1 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:21.795403+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:21.795258+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:49:17.636705+0000 3 0 2024-09-04T19:28:31.526 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-09-04T19:28:16.847001+0000 32'64 40:135 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:29:12.801393+0000 2 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-09-04T19:28:19.779203+0000 30'62 42:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:40:49.819057+0000 2 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:09.682416+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:18:13.686845+0000 2 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:21.839415+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:21.839199+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:36:25.772530+0000 3 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 0 0 0 1813 0 0 18 48 18 active+clean 2024-09-04T19:28:27.264987+0000 37'66 42:124 [1,0,2] 1 [1,0,2] 1 37'66 2024-09-04T19:28:25.765355+0000 37'66 2024-09-04T19:28:25.765355+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:28:22.847668+0000 32'63 42:131 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:22.847326+0000 32'63 2024-09-04T19:28:22.847326+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:54:36.523810+0000 4 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:20.712788+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T07:00:53.629583+0000 3 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 0 0 0 1234 0 0 15 48 15 active+clean 2024-09-04T19:28:15.945931+0000 37'63 40:124 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:00:22.584269+0000 3 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-09-04T19:28:20.870966+0000 30'62 42:123 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:52:15.754481+0000 1 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:28:21.610326+0000 32'64 42:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:15.727617+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:43:04.509732+0000 3 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:28:23.856089+0000 32'63 40:135 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:23.855894+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:04:51.678455+0000 3 0 2024-09-04T19:28:31.527 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:28:27.180901+0000 32'64 42:125 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:23.733961+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:13:08.553257+0000 4 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 1 0 0 0 0 617 0 0 14 48 14 active+clean 2024-09-04T19:28:21.612474+0000 30'62 42:129 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:49:04.710621+0000 1 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:28:18.661155+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:19:10.041108+0000 3 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1234 0 0 14 48 14 active+clean 2024-09-04T19:28:22.938490+0000 30'62 40:123 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:22.938216+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:53:16.409976+0000 2 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:28:21.743932+0000 32'63 42:135 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:21.743651+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:40:33.115580+0000 4 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 6 0 0 0 0 3435 0 0 18 48 18 active+clean 2024-09-04T19:28:23.764072+0000 38'66 42:133 [1,2,0] 1 [1,2,0] 1 38'66 2024-09-04T19:28:23.763852+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:29:57.764591+0000 6 0 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:28:31.528 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:28:31.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-04T19:28:31.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:28:31.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:28:31.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:28:31.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:28:31.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:28:31.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:28:31.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-04T19:28:31.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-04T19:28:32.404 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:28:32.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-04T19:28:32.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-04T19:28:32.994 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:28:33.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-04T19:28:33.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-04T19:28:33.541 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:28:33.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:33.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-04T19:28:34.109 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-04T19:28:34.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:34.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-04T19:28:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-04T19:28:35.058 INFO:tasks.workunit.client.0.smithi174.stdout:bringing up non_divergent 0 2 2024-09-04T19:28:35.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-04T19:28:35.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:24.826029+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:24.825823+0000 32'63 2024-09-04T19:28:24.825823+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:14:56.467872+0000 3 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:27.265412+0000 32'64 42:117 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:22.707656+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:19.796792+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:32:31.482011+0000 4 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:25.794128+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:25.793935+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:08:42.989229+0000 3 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 3 0 0 0 0 1851 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:24.795995+0000 30'62 42:123 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:24.795794+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:04:35.201731+0000 3 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 5 0 0 0 0 3435 0 0 17 48 17 stale+active+clean 2024-09-04T19:28:24.868232+0000 37'65 40:137 [2,0,1] 2 [2,0,1] 2 37'65 2024-09-04T19:28:24.868034+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:12:52.349731+0000 5 0 2024-09-04T19:28:35.468 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:10.687604+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:54:21.986691+0000 1 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:26.855832+0000 30'62 42:129 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:26.855615+0000 30'62 2024-09-04T19:28:26.855615+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:35:21.651481+0000 2 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 0 0 0 2430 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:27.265000+0000 32'64 42:117 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:26.794555+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:18.874124+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:27:06.152174+0000 6 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:11.700144+0000 30'62 42:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:42:58.767703+0000 2 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:21.611014+0000 32'63 42:130 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:07:07.593683+0000 4 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:19.797062+0000 32'63 42:135 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:40:44.905826+0000 1 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:21.795403+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:21.795258+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:49:17.636705+0000 3 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:16.847001+0000 32'64 40:135 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:29:12.801393+0000 2 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:19.779203+0000 30'62 42:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:40:49.819057+0000 2 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:09.682416+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:18:13.686845+0000 2 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:21.839415+0000 32'64 40:136 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:21.839199+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:36:25.772530+0000 3 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 0 0 0 1813 0 0 18 48 18 stale+active+clean 2024-09-04T19:28:27.264987+0000 37'66 42:124 [1,0,2] 1 [1,0,2] 1 37'66 2024-09-04T19:28:25.765355+0000 37'66 2024-09-04T19:28:25.765355+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:22.847668+0000 32'63 42:131 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:22.847326+0000 32'63 2024-09-04T19:28:22.847326+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:54:36.523810+0000 4 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:20.712788+0000 32'63 42:135 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T07:00:53.629583+0000 3 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 0 0 0 1234 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:15.945931+0000 37'63 40:124 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:00:22.584269+0000 3 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 1 0 0 0 0 617 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:20.870966+0000 30'62 42:123 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:52:15.754481+0000 1 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:21.610326+0000 32'64 42:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:15.727617+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:43:04.509732+0000 3 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1851 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:23.856089+0000 32'63 40:135 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:23.855894+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:04:51.678455+0000 3 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2468 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:27.180901+0000 32'64 42:125 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:23.733961+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:13:08.553257+0000 4 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 1 0 0 0 0 617 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:21.612474+0000 30'62 42:129 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:49:04.710621+0000 1 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 3 0 0 0 0 1851 0 0 16 48 16 stale+active+clean 2024-09-04T19:28:18.661155+0000 32'64 42:136 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:19:10.041108+0000 3 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1234 0 0 14 48 14 stale+active+clean 2024-09-04T19:28:22.938490+0000 30'62 40:123 [2,0,1] 2 [2,0,1] 2 30'62 2024-09-04T19:28:22.938216+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:53:16.409976+0000 2 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 4 0 0 0 0 2468 0 0 15 48 15 stale+active+clean 2024-09-04T19:28:21.743932+0000 32'63 42:135 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:21.743651+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:40:33.115580+0000 4 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 6 0 0 0 0 3435 0 0 18 48 18 stale+active+clean 2024-09-04T19:28:23.764072+0000 38'66 42:133 [1,2,0] 1 [1,2,0] 1 38'66 2024-09-04T19:28:23.763852+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:29:57.764591+0000 6 0 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:28:35.470 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:28:35.471 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:28:35.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:28:35.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:28:35.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:28:35.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:28:35.489 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:28:35.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:28:35.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-04T19:28:35.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:28:35.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:28:35.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:28:35.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:28:35.526 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:35.526+0000 7fb0375da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:35.528 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:35.526+0000 7fb0375da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:35.529 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:35.530+0000 7fb0375da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:28:35.966 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:28:35.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:35.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:28:36.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:36.556 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:36.554+0000 7fb0375da7c0 -1 Falling back to public interface 2024-09-04T19:28:37.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:37.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:37.394 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:28:37.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:28:37.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:37.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:28:37.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:38.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:38.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:38.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:28:38.880 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:28:38.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:38.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:28:39.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:39.954 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:39.954+0000 7fb0375da7c0 -1 osd.0 36 log_to_monitors true 2024-09-04T19:28:40.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:40.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:40.321 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:28:40.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:28:40.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:40.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:28:40.837 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up out weight 0 up_from 56 up_thru 35 down_at 46 last_clean_interval [5,36) [v2:127.0.0.1:6802/3656998195,v1:127.0.0.1:6803/3656998195] [v2:127.0.0.1:6804/3656998195,v1:127.0.0.1:6805/3656998195] exists,up e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:28:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:28:40.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:28:40.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:28:40.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:28:40.844 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:28:40.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:28:40.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-04T19:28:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:28:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:28:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:28:40.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:28:40.882 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:40.882+0000 7fe3af4397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:40.903 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:40.902+0000 7fe3af4397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:40.904 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:40.902+0000 7fe3af4397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:28:41.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:41.293 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:28:41.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:28:41.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:41.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:28:41.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:42.213 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:42.210+0000 7fe3af4397c0 -1 Falling back to public interface 2024-09-04T19:28:42.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:42.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:42.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:28:42.742 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:28:42.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:42.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:28:43.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:44.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:44.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:44.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:28:44.272 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:28:44.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:44.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:28:44.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:28:45.480 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:28:45.478+0000 7fe3af4397c0 -1 osd.2 36 log_to_monitors true 2024-09-04T19:28:45.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:28:45.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:28:45.764 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:28:45.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:28:45.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:28:45.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:28:46.211 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 61 up_thru 33 down_at 49 last_clean_interval [14,36) [v2:127.0.0.1:6810/310914855,v1:127.0.0.1:6811/310914855] [v2:127.0.0.1:6812/310914855,v1:127.0.0.1:6813/310914855] exists,up bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:28:46.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:28:46.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:28:46.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:28:46.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:46.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-04T19:28:46.834 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already in. 2024-09-04T19:28:46.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:28:46.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-04T19:28:47.822 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-04T19:28:47.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:28:47.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-04T19:28:47.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:28:47.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:28:47.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:28:47.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:28:47.844 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:28:47.844 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:28:47.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:28:47.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:28:47.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:28:47.911 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:28:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:28:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:28:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:28:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:28:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:48.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:28:48.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168580 2024-09-04T19:28:48.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168580 2024-09-04T19:28:48.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-240518168580' 2024-09-04T19:28:48.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:48.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:28:48.784 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:28:48.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:28:48.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:28:48.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:28:48.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:28:48.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:28:48.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005059 2024-09-04T19:28:48.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005059 2024-09-04T19:28:48.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-240518168580 2-261993005059' 2024-09-04T19:28:48.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:28:48.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-240518168580 2024-09-04T19:28:48.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:28:48.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:28:49.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-240518168580 2024-09-04T19:28:49.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:28:49.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168580 2024-09-04T19:28:49.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 240518168580' 2024-09-04T19:28:49.002 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 240518168580 2024-09-04T19:28:49.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:28:49.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168578 -lt 240518168580 2024-09-04T19:28:49.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:28:50.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:28:50.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:28:50.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168580 -lt 240518168580 2024-09-04T19:28:50.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:28:50.914 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:28:50.914 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-261993005059 2024-09-04T19:28:50.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:28:50.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-261993005059 2024-09-04T19:28:50.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:28:50.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005059 2024-09-04T19:28:50.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 261993005059' 2024-09-04T19:28:50.918 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 261993005059 2024-09-04T19:28:50.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:28:51.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005059 -lt 261993005059 2024-09-04T19:28:51.375 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:28:51.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:51.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:51.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-04T19:28:51.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:51.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:52.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:28:52.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:52.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:52.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:28:52.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:28:52.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:28:52.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:52.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:52.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:52.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:52.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:52.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:52.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:53.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:28:53.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:53.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:53.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:28:54.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:28:54.011 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:28:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:28:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:28:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:28:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:28:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:28:54.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:28:54.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:54.747 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:54.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:54.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:54.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:54.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:54.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:55.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:28:55.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:55.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:55.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:55.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:28:55.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:28:55.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:28:55.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:28:55.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:28:55.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:28:55.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:28:55.715 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:28:55.715 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:28:56.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:28:56.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:28:56.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:28:56.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:28:56.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:28:56.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:28:56.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:56.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:56.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:56.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:56.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:56.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:56.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:57.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:28:57.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:57.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:57.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:28:57.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:28:57.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:28:57.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:28:58.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:28:58.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:28:58.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:28:58.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:28:58.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:28:59.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:28:59.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:28:59.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:28:59.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:28:59.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:28:59.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:28:59.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:28:59.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:28:59.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:28:59.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:28:59.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:28:59.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:00.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:00.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:00.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:00.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:00.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:00.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:29:00.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:00.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:29:02.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:02.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:02.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:02.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:02.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:02.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:02.156 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:02.156 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:02.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:02.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:02.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:02.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:03.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:03.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:03.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:29:03.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:03.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:29:06.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:06.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:06.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:07.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:07.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:07.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:07.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:07.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:07.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:07.798 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:07.798 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:08.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:08.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:08.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:29:08.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:08.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:29:14.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:14.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:14.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:14.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:14.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:14.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:14.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:14.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:15.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:15.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:15.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:15.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:15.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:15.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:15.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:15.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:16.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:16.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:16.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:29:16.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:16.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:29:23.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:23.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:23.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:23.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:23.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:23.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:23.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:23.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:24.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:24.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:24.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:24.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:24.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:24.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:24.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:25.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:25.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:25.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:29:25.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:29:25.613 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:29:25.613 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "3bf416f0-0778-43ad-a9bf-cc2ba8281935", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:29:25.612875+0000", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:29:25.614 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:29:25.615 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 32 pgs degraded", 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 32 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.616 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.617 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.618 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.619 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.620 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.621 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.622 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.623 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.624 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:25:58.160929Z", 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:25:58.160929Z", 2024-09-04T19:29:25.625 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:29:25.626 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.627 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.628 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 69, 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:25:59.047144+0000", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:28:49.024828+0000", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:28:45.777987+0000", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:28:47.490623+0000", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:29:25.629 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 9, 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:29:25.630 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:27:43.845295+0000", 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 1, 2024-09-04T19:29:25.631 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool", 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 0, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "on", 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 32, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 18, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 32, 2024-09-04T19:29:25.632 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 32, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 32, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "68", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "31", 2024-09-04T19:29:25.633 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "selfmanaged", 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [], 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:29:25.634 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "", 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:29:25.635 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 0, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {}, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "read_balance": { 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "score_type": "Fair distribution", 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "score_acting": 1.7799999713897705, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "score_stable": 1.7799999713897705, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_score": 0.67000001668930054, 2024-09-04T19:29:25.636 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_acting": 1.190000057220459, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_stable": 1.190000057220459, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity_weighted": 1, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity": 1, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity_weighted": 1 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "e58308e4-8e6d-41b5-8d4b-0f1787630e30", 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:29:25.637 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 36, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 56, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 68, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 46, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.638 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.639 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:29:25.640 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:29:25.641 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/3656998195", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/3656998195", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3656998195", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3656998195", 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:29:25.642 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "2e03be93-3167-48bf-8313-79cddea604a4", 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:29:25.643 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 44, 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 48, 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.644 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.645 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:29:25.646 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:29:25.647 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/978199495", 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/978199495", 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/978199495", 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/978199495", 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "bb73362d-2836-4717-a60d-80254994227d", 2024-09-04T19:29:25.648 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 14, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 36, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 61, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 66, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 49, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.649 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:29:25.650 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.651 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.652 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/310914855", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/310914855", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/310914855", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/310914855", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.653 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:32.229092+0000", 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:26:32.315422+0000", 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.654 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:32.845883+0000", 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:27:06.506399+0000", 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:33.370949+0000", 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:29:25.655 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:27:40.621183+0000", 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:29:25.656 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:29:25.657 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/3656998195,v1:127.0.0.1:6805/3656998195]", 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:29:25.658 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:29:25.659 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:26:05.594427Z", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:29:25.660 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/3656998195,v1:127.0.0.1:6803/3656998195]", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3656998195,v1:127.0.0.1:6809/3656998195]", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3656998195,v1:127.0.0.1:6807/3656998195]", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:29:25.661 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/978199495,v1:127.0.0.1:6813/978199495]", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:29:25.662 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:29:25.663 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:26:39.407268Z", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/978199495,v1:127.0.0.1:6811/978199495]", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:29:25.664 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/978199495,v1:127.0.0.1:6817/978199495]", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/978199495,v1:127.0.0.1:6815/978199495]", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:29:25.665 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/310914855,v1:127.0.0.1:6813/310914855]", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:29:25.666 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:27:13.213681Z", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:29:25.667 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/310914855,v1:127.0.0.1:6811/310914855]", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/310914855,v1:127.0.0.1:6817/310914855]", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/310914855,v1:127.0.0.1:6815/310914855]", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:29:25.668 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:29:25.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 57 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 61 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.670 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 69, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:29:25.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.672 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.673 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:29:25.674 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:29:25.675 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stderr:report 2722184570 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:29:25.676 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.677 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.678 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:29:25.679 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.680 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:29:25.681 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.682 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:29:25.683 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.684 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:29:25.685 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 0, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:25:59:023174+0000", 2024-09-04T19:29:25.686 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.687 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:29:25.688 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 32, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 32, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62046, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 102, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 306, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 102, 2024-09-04T19:29:25.689 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 102, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 75, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 75, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:29:25.690 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:29:25.691 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:29:25.692 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 486, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 486, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 64, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 64, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:29:25.693 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 96, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 84260, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2648, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81525, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314488540, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036264960, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2711552, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 671740, 2024-09-04T19:29:25.694 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83482468 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:29:25.695 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 112, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 112, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 112000000, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 112000000 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 96, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:29:25.696 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 84260, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2648, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81525, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314488540, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036264960, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2711552, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 671740, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:29:25.697 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83482468 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 112, 2024-09-04T19:29:25.698 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 112, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 112000000, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 112000000 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 32, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62046, 2024-09-04T19:29:25.699 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 102, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 306, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 102, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 102, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 75, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 75, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:29:25.700 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:29:25.701 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:29:25.702 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1220608, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 184566, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 486, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 486, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 64, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 64, 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:29:25.703 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 240518168587 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672983 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 261993005066 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.704 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 32 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 19, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 32, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 13, 2024-09-04T19:29:25.705 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 32, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.706 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 5, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:28:36.547841+0000", 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 92, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 427, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 579, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:29:25.707 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-04T19:29:25.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-04T19:29:25.709 INFO:tasks.workunit.client.0.smithi174.stdout:writing non-divergent object existing_4 2024-09-04T19:29:25.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 69:135 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 69:114 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 69:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049083+0000 0 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 69:101 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 69:122 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.004 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 69:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049203+0000 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 69:135 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049163+0000 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 69:132 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 69:134 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049224+0000 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 69:131 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049183+0000 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 69:126 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 32'64 69:114 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 69:131 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:29:26.005 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 69:118 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049143+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 69:101 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:29:06.066738+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 69:115 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049106+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 69:119 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049125+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 69:124 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 69:128 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049263+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 69:112 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 69:114 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 69:113 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 69:119 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049245+0000 0 0 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:29:26.006 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:29:26.007 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:29:26.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-04T19:29:26.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:29:26.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-04T19:29:26.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:29:26.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:29:26.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:29:26.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:29:26.163 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:29:26.163 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:29:26.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:29:26.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:29:26.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:29:26.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:29:26.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:29:26.726 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:29:26.727 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:29:26.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:29:26.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:29:26.727 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:29:26.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168590 2024-09-04T19:29:26.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168590 2024-09-04T19:29:26.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-240518168590' 2024-09-04T19:29:26.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:29:26.897 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:29:27.045 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:29:27.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:29:27.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:29:27.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:29:27.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:29:27.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005069 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005069 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-240518168590 2-261993005069' 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-240518168590 2024-09-04T19:29:27.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:29:27.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:29:27.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-240518168590 2024-09-04T19:29:27.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:29:27.205 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 240518168590 2024-09-04T19:29:27.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168590 2024-09-04T19:29:27.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 240518168590' 2024-09-04T19:29:27.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:29:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168588 -lt 240518168590 2024-09-04T19:29:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:29:28.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:29:28.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:29:29.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168590 -lt 240518168590 2024-09-04T19:29:29.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:29:29.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-261993005069 2024-09-04T19:29:29.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:29:29.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:29:29.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-261993005069 2024-09-04T19:29:29.028 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:29:29.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005069 2024-09-04T19:29:29.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 261993005069' 2024-09-04T19:29:29.029 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 261993005069 2024-09-04T19:29:29.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:29:29.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005069 -lt 261993005069 2024-09-04T19:29:29.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:29:29.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:29.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:30.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-04T19:29:30.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:30.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:30.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:30.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:30.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:30.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:30.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:30.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:30.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:30.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:30.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:30.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:30.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:29:30.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:29:30.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:29:30.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:30.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:29:31.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:31.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:31.092 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:31.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:31.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:31.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:31.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:31.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:31.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:31.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:31.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:31.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:32.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:32.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:32.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:32.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:32.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:29:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:29:32.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:32.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:32.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:33.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:33.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:33.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:33.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:33.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:33.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:33.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:34.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:34.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:34.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:29:34.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:34.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:29:34.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:34.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:34.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:35.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:35.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:35.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:35.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:35.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:35.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:35.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:35.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:35.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:35.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:35.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:35.686 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:35.686 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:36.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:36.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:36.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:29:36.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:36.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:29:37.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:37.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:37.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:37.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:37.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:37.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:37.017 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:37.017 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:37.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:37.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:37.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:37.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:37.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:37.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:37.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:37.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:37.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:37.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:37.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:37.996 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:37.996 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:29:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:29:40.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:40.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:40.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:40.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:40.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:40.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:40.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:40.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:40.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:40.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:40.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:40.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:41.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:41.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:41.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:41.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:41.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:41.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:29:41.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:41.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:29:44.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:44.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:44.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:44.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:44.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:44.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:44.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:44.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:45.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:45.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:45.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:45.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:45.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:46.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:46.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:46.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:29:46.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:46.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:29:52.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:29:52.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:29:52.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:29:52.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:29:52.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:29:52.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:29:52.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:29:52.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:29:53.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:29:53.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:29:53.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:29:53.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:29:53.778 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:29:53.779 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:29:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:29:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:29:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:29:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:29:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:30:01.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:30:01.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:30:01.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:30:02.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:30:02.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:30:02.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:30:02.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:30:02.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:30:02.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:30:02.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:30:02.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:30:03.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:30:03.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:30:03.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:30:03.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:30:03.661 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "3bf416f0-0778-43ad-a9bf-cc2ba8281935", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:30:03.661002+0000", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:30:03.662 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.663 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 64 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.664 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.2 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.3 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.4 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.665 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.5 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.6 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.7 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.8 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.9 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.666 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.d is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.e is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.667 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.f is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.10 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.12 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.668 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.13 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.14 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.15 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.16 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.17 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.669 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.18 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.19 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.670 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1d is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1e is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.1f is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.671 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:30:03.672 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:25:58.160929Z", 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:25:58.160929Z", 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:30:03.673 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:30:03.674 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.675 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:30:03.676 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 69, 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:25:59.047144+0000", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:28:49.024828+0000", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:28:45.777987+0000", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:28:47.490623+0000", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:30:03.677 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 9, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:30:03.678 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:27:43.845295+0000", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 1, 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool", 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:30:03.679 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 0, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "on", 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 32, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 18, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 32, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 32, 2024-09-04T19:30:03.680 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 32, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "68", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "31", 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:30:03.681 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "selfmanaged", 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [], 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:30:03.682 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "", 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:30:03.683 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 0, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {}, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "read_balance": { 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "score_type": "Fair distribution", 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "score_acting": 1.7799999713897705, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "score_stable": 1.7799999713897705, 2024-09-04T19:30:03.684 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_score": 0.67000001668930054, 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_acting": 1.190000057220459, 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "raw_score_stable": 1.190000057220459, 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity_weighted": 1, 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity": 1, 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "average_primary_affinity_weighted": 1 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.685 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "e58308e4-8e6d-41b5-8d4b-0f1787630e30", 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:30:03.686 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 36, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 56, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 68, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 46, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.687 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.688 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.689 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:30:03.690 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3656998195 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/3656998195", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/3656998195", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3656998195", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3656998195", 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:30:03.691 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "2e03be93-3167-48bf-8313-79cddea604a4", 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:30:03.692 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 44, 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 48, 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.693 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.694 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.695 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:30:03.696 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 978199495 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/978199495", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/978199495", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/978199495", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/978199495", 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:30:03.697 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "bb73362d-2836-4717-a60d-80254994227d", 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 14, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 36, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 61, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 66, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 49, 2024-09-04T19:30:03.698 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.699 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.700 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.701 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:30:03.702 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 310914855 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/310914855", 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/310914855", 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/310914855", 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/310914855", 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.703 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:32.229092+0000", 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:26:32.315422+0000", 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:32.845883+0000", 2024-09-04T19:30:03.704 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:27:06.506399+0000", 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:28:33.370949+0000", 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:30:03.705 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:27:40.621183+0000", 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:30:03.706 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:30:03.707 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/3656998195,v1:127.0.0.1:6805/3656998195]", 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:30:03.708 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:30:03.709 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:26:05.594427Z", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:30:03.710 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/3656998195,v1:127.0.0.1:6803/3656998195]", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3656998195,v1:127.0.0.1:6809/3656998195]", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3656998195,v1:127.0.0.1:6807/3656998195]", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:30:03.711 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:30:03.712 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/978199495,v1:127.0.0.1:6813/978199495]", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:30:03.713 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.714 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:26:39.407268Z", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/978199495,v1:127.0.0.1:6811/978199495]", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/978199495,v1:127.0.0.1:6817/978199495]", 2024-09-04T19:30:03.715 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/978199495,v1:127.0.0.1:6815/978199495]", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:30:03.716 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/310914855,v1:127.0.0.1:6813/310914855]", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:30:03.717 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:30:03.718 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:27:13.213681Z", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/310914855,v1:127.0.0.1:6811/310914855]", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/310914855,v1:127.0.0.1:6817/310914855]", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/310914855,v1:127.0.0.1:6815/310914855]", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:30:03.719 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.720 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:30:03.721 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 57 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.722 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 61 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 69, 2024-09-04T19:30:03.723 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:30:03.724 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.725 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:30:03.726 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.727 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:30:03.727 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:30:03.727 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.727 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.727 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:30:03.728 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.729 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:30:03.730 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:30:03.730 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.730 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.730 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:30:03.730 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:30:03.731 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.731 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.731 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:30:03.731 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.732 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.733 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:30:03.734 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:30:03.735 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.736 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:30:03.737 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.738 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:30:03.739 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:30:03.740 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:30:03.741 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:30:03.742 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:30:03.743 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.744 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:30:03.745 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:30:03.746 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:30:03.747 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 0, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:30:03.748 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:30:03.749 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:25:59:023174+0000", 2024-09-04T19:30:03.749 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:30:03.749 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:30:03.749 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:30:03.749 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:30:03.750 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:30:03.750 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:30:03.750 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:30:03.750 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:30:03.750 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.751 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:30:03.752 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 32, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 32, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:30:03.753 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62396, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 102, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 306, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 102, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 102, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:30:03.754 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 76, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 76, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:30:03.755 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:30:03.756 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:30:03.757 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:30:03.758 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 487, 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 487, 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 64, 2024-09-04T19:30:03.759 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 64, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 96, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:30:03.760 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 84260, 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2648, 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81525, 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314488540, 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:30:03.761 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036264960, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2711552, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 672440, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:30:03.762 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83482468 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 112, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 112, 2024-09-04T19:30:03.763 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 112000000, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 112000000 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 96, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:30:03.764 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 84260, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2648, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81525, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314488540, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036264960, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2711552, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 672440, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:30:03.765 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83482468 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 112, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 112, 2024-09-04T19:30:03.766 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 112000000, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 112000000 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 32, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62396, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 102, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 306, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:30:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 102, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 102, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 76, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 76, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:30:03.768 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:30:03.769 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:30:03.770 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1220608, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 185266, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:30:03.771 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 487, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 487, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 64, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 64, 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.772 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 240518168597 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672983 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 261993005076 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:30:03.773 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 32 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 19, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 32, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 13, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 32, 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:30:03.774 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:30:03.775 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 7, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:29:48.574465+0000", 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 111, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 496, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:30:03.776 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 667, 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stderr:report 3627582900 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:30:03.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-04T19:30:03.778 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery 2024-09-04T19:30:03.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 69:135 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 69:114 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 69:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049083+0000 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 69:101 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 69:122 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 69:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049203+0000 0 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 69:135 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:04.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049163+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 69:132 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 69:134 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049224+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 69:131 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049183+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 69:126 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 69:115 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:29:28.580111+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 69:131 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 69:118 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049143+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 69:101 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:29:06.066738+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 69:115 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049106+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 69:119 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049125+0000 0 0 2024-09-04T19:30:04.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 69:124 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 69:128 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049263+0000 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 69:112 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 69:114 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 69:113 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 69:119 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049245+0000 0 0 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:04.087 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:04.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:30:04.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-04T19:30:04.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:30:04.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:30:04.102 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:04.102 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:04.102 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:04.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:30:04.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:30:04.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok set_recovery_delay 100000 2024-09-04T19:30:04.208 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:30:04.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-04T19:30:04.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-04T19:30:04.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:30:04.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:30:04.222 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:04.222 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:04.222 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:04.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:30:04.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:30:04.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok set_recovery_delay 100000 2024-09-04T19:30:04.336 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:30:04.350 INFO:tasks.workunit.client.0.smithi174.stdout:revive divergent 1 2024-09-04T19:30:04.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-04T19:30:04.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:04.752 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:04.752 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 69:135 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:04.752 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 69:114 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.752 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 69:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049083+0000 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 69:101 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 69:122 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 69:128 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049203+0000 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 69:135 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049163+0000 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 69:132 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 69:134 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 69:118 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049224+0000 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 69:131 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 69:106 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049183+0000 0 0 2024-09-04T19:30:04.753 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 69:126 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 69:115 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:29:28.580111+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 69:131 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 69:118 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049143+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 69:101 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:29:06.066738+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 69:115 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049106+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 69:125 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 69:119 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049125+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 69:124 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 69:128 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049263+0000 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 69:112 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 69:114 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.754 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 69:112 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.755 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 69:113 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:04.755 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 69:119 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:28:54.049245+0000 0 0 2024-09-04T19:30:04.755 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:04.755 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:04.755 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:04.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-04T19:30:05.532 INFO:tasks.workunit.client.0.smithi174.stderr:noup is set 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:30:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:30:05.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:05.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:30:05.557 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:30:05.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:30:05.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:30:05.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:30:05.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:30:05.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:30:05.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:30:05.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:30:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:05.601+0000 7f80c43ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:05.605+0000 7f80c43ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:05.608 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:05.605+0000 7f80c43ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:06.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-04T19:30:06.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-04T19:30:06.613 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:06.609+0000 7f80c43ff7c0 -1 Falling back to public interface 2024-09-04T19:30:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:09.785+0000 7f80c43ff7c0 -1 osd.1 45 log_to_monitors true 2024-09-04T19:30:11.004 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery divergent 2024-09-04T19:30:11.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-04T19:30:11.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 71:138 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 71:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.403006+0000 0 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 71:139 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 70:120 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.005957+0000 0 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 71:104 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402965+0000 0 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 71:128 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402958+0000 0 0 2024-09-04T19:30:11.419 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 71:125 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 71:115 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402940+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006190+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 71:138 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 70:108 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006115+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 71:135 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 70:136 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 70:120 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006227+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 71:134 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 70:108 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006152+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 70:128 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 71:128 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402989+0000 0 0 2024-09-04T19:30:11.420 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 71:118 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.403014+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 71:134 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 70:120 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006077+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 70:103 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:08.015355+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 70:117 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006001+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 71:128 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402981+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 70:121 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006040+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 71:127 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402973+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 70:130 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006302+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 71:115 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402949+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 71:117 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402997+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 71:115 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402932+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 71:116 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:09.402921+0000 0 0 2024-09-04T19:30:11.421 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 70:121 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:10.006264+0000 0 0 2024-09-04T19:30:11.422 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:11.422 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:11.422 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:11.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-04T19:30:11.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:30:11.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:30:11.436 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:11.437 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:11.437 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:11.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:30:11.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:30:11.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok set_recovery_delay 100000 2024-09-04T19:30:11.563 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:30:11.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-04T19:30:12.355 INFO:tasks.workunit.client.0.smithi174.stderr:noup is unset 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:12.378 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:12.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:12.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:30:13.046 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 56 up_thru 68 down_at 46 last_clean_interval [5,36) [v2:127.0.0.1:6802/3656998195,v1:127.0.0.1:6803/3656998195] [v2:127.0.0.1:6804/3656998195,v1:127.0.0.1:6805/3656998195] exists,up e58308e4-8e6d-41b5-8d4b-0f1787630e30 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:13.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:13.049 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:13.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:13.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:13.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:30:13.724 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 74 up_thru 74 down_at 48 last_clean_interval [10,45) [v2:127.0.0.1:6818/3193162851,v1:127.0.0.1:6819/3193162851] [v2:127.0.0.1:6820/3193162851,v1:127.0.0.1:6821/3193162851] exists,up 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:13.726 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:30:14.245 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 61 up_thru 74 down_at 49 last_clean_interval [14,36) [v2:127.0.0.1:6810/310914855,v1:127.0.0.1:6811/310914855] [v2:127.0.0.1:6812/310914855,v1:127.0.0.1:6813/310914855] exists,up bb73362d-2836-4717-a60d-80254994227d 2024-09-04T19:30:14.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:30:14.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:30:14.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:30:14.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 72:140 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 72:119 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421949+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 72:141 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085933+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 72:106 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421848+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421825+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 72:127 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421782+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 72:134 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086068+0000 0 0 2024-09-04T19:30:14.654 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 72:140 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086025+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 72:137 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 72:140 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086090+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 72:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086046+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 72:132 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421909+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 72:120 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421973+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 72:136 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 72:124 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086002+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 72:107 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:13.034508+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 72:121 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085960+0000 0 0 2024-09-04T19:30:14.655 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421888+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 72:125 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085981+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 72:129 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421869+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 72:134 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086131+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 72:117 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421802+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 72:119 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421929+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421762+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 72:118 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421737+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 72:125 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086111+0000 0 0 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:14.670 INFO:tasks.workunit.client.0.smithi174.stdout:wait for peering 2024-09-04T19:30:14.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-04T19:30:14.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:15.084 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:15.084 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 72:140 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:15.084 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 72:119 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421949+0000 0 0 2024-09-04T19:30:15.084 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 72:141 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085933+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 72:106 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421848+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421825+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 72:127 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421782+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 72:134 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086068+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 72:140 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086025+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 72:137 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 72:140 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086090+0000 0 0 2024-09-04T19:30:15.085 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 72:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086046+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 72:132 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421909+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 72:120 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421973+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 72:136 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 72:124 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086002+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 72:107 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:13.034508+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 72:121 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085960+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421888+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 72:125 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085981+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 72:129 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421869+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 72:134 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086131+0000 0 0 2024-09-04T19:30:15.086 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 72:117 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421802+0000 0 0 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 72:119 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421929+0000 0 0 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421762+0000 0 0 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 72:118 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421737+0000 0 0 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 72:125 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086111+0000 0 0 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:15.087 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:15.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-04T19:30:15.206 INFO:tasks.workunit.client.0.smithi174.stdout:killing divergent 1 2024-09-04T19:30:15.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-04T19:30:15.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.750189+0000 32'63 72:140 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:20.885005+0000 32'63 2024-09-04T19:28:20.885005+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:11:05.518217+0000 3 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749146+0000 32'64 72:119 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421949+0000 0 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 4 0 0 2468 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749319+0000 32'64 72:141 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:22:40.127978+0000 4 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270480+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085933+0000 0 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751010+0000 30'62 72:106 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421848+0000 0 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750417+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421825+0000 0 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 3 0 0 1851 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750364+0000 32'64 72:127 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T02:53:19.292685+0000 1 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.748924+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421782+0000 0 0 2024-09-04T19:30:15.614 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272227+0000 32'64 72:134 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086068+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 6 0 0 3702 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750156+0000 32'64 72:140 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:05:07.193965+0000 6 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 2 0 0 1196 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271374+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086025+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 5 0 0 3085 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749417+0000 32'63 72:137 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T20:28:51.182668+0000 4 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 1 0 0 617 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.273673+0000 32'63 72:140 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:31:37.850158+0000 1 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 3 0 0 1813 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270708+0000 32'63 72:124 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086090+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 2 0 0 1234 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.751171+0000 32'64 72:136 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:04:49.850070+0000 2 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.271929+0000 30'62 72:112 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086046+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 3 0 0 1851 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270926+0000 32'63 72:132 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:47:22.164295+0000 2 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 4 0 0 2430 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749903+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421909+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 0 4 0 0 2780 0 0 17 48 17 active+undersized+degraded 2024-09-04T19:28:48.748925+0000 69'65 72:120 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421973+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 4 0 0 2468 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.664688+0000 32'63 72:136 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:17.903526+0000 32'63 2024-09-04T19:28:17.903526+0000 0 1 periodic scrub scheduled @ 2024-09-06T03:02:02.946655+0000 4 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.270876+0000 32'63 72:124 [0,2] 0 [0,2] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086002+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:29:04.664976+0000 30'62 72:107 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:13.034508+0000 0 0 2024-09-04T19:30:15.615 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.270689+0000 30'62 72:121 [0,2] 0 [0,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085960+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.750041+0000 32'64 72:130 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421888+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.271820+0000 32'64 72:125 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.085981+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.749247+0000 32'63 72:129 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421869+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.274433+0000 32'64 72:134 [0,2] 0 [0,2] 0 32'64 2024-09-04T19:28:17.629351+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086131+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.750820+0000 30'62 72:117 [2,0] 2 [2,0] 2 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421802+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.749657+0000 32'64 72:119 [2,0] 2 [2,0] 2 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421929+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 2 0 0 1195 0 0 14 48 14 active+undersized+degraded 2024-09-04T19:28:48.751456+0000 30'62 72:117 [2,0] 2 [2,0] 2 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421762+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 3 0 0 1812 0 0 15 48 15 active+undersized+degraded 2024-09-04T19:28:48.752215+0000 32'63 72:118 [2,0] 2 [2,0] 2 32'63 2024-09-04T19:28:16.657040+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:14.421737+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 0 4 0 0 2429 0 0 16 48 16 active+undersized+degraded 2024-09-04T19:28:48.272856+0000 32'64 72:125 [0,2] 0 [0,2] 0 0'0 2024-09-04T19:27:43.895503+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:15.086111+0000 0 0 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:15.616 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:15.617 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:15.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-04T19:30:15.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:30:15.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:30:15.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:30:15.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:30:15.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27364.out 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27364.out 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-04T19:30:15.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27364.out 2024-09-04T19:30:17.341 INFO:tasks.workunit.client.0.smithi174.stderr:Exporting 1.1b info 1.1b( v 32'64 lc 0'0 (29'48,32'64] local-lis/les=74/75 n=4 ec=31/22 lis/c=74/37 les/c/f=75/38/0 sis=74) 2024-09-04T19:30:17.342 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-04T19:30:17.348 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-04T19:30:17.348 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-04T19:30:17.355 INFO:tasks.workunit.client.0.smithi174.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-04T19:30:17.356 INFO:tasks.workunit.client.0.smithi174.stderr:Export successful 2024-09-04T19:30:17.357 INFO:tasks.workunit.client.0.smithi174.stdout: marking collection for removal 2024-09-04T19:30:17.357 INFO:tasks.workunit.client.0.smithi174.stdout:setting '_remove' omap key 2024-09-04T19:30:17.357 INFO:tasks.workunit.client.0.smithi174.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-04T19:30:17.449 INFO:tasks.workunit.client.0.smithi174.stderr:Remove successful 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27364.out 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-04T19:30:17.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27364.out 2024-09-04T19:30:19.240 INFO:tasks.workunit.client.0.smithi174.stdout:get_pg_num_history pg_num_history pg_num_history(e77 pg_nums {1={22=1,31=32}} deleted_pools ) 2024-09-04T19:30:19.240 INFO:tasks.workunit.client.0.smithi174.stdout:Importing pgid 1.1b 2024-09-04T19:30:19.240 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-04T19:30:19.240 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:30:19.311 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-04T19:30:19.311 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:30:19.344 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-04T19:30:19.345 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:30:19.378 INFO:tasks.workunit.client.0.smithi174.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-04T19:30:19.378 INFO:tasks.workunit.client.0.smithi174.stdout:snapset 0=[]:{} 2024-09-04T19:30:19.411 INFO:tasks.workunit.client.0.smithi174.stdout:write_pg epoch 75 info 1.1b( v 32'64 lc 0'0 (29'48,32'64] local-lis/les=74/75 n=4 ec=31/22 lis/c=74/37 les/c/f=75/38/0 sis=74) 2024-09-04T19:30:19.445 INFO:tasks.workunit.client.0.smithi174.stdout:Import successful 2024-09-04T19:30:19.988 INFO:tasks.workunit.client.0.smithi174.stdout:reviving divergent 1 2024-09-04T19:30:19.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-04T19:30:19.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:13.977362+0000 32'63 77:154 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:24.825823+0000 32'63 2024-09-04T19:28:24.825823+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:13:13.378565+0000 3 0 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:13.487229+0000 32'64 77:144 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:29.804962+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745702+0000 0 0 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:30:13.976701+0000 32'64 77:155 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:45:26.569000+0000 4 0 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:13.487164+0000 32'63 77:141 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:25.793935+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745676+0000 0 0 2024-09-04T19:30:20.396 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:13.488089+0000 30'62 77:122 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:24.795794+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745729+0000 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:13.489778+0000 32'64 77:135 [1,2,0] 1 [1,2,0] 1 37'65 2024-09-04T19:28:24.868034+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:30:13.444639+0000 32'64 77:165 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:13:13.551852+0000 1 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:13.442893+0000 30'62 77:143 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:26.855615+0000 30'62 2024-09-04T19:28:26.855615+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745563+0000 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:30:13.441275+0000 32'64 77:147 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:30.860452+0000 32'64 2024-09-04T19:28:30.860452+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745490+0000 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-09-04T19:30:13.975998+0000 32'64 77:154 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:03:23.402239+0000 6 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:13.488341+0000 30'62 77:124 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745754+0000 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-09-04T19:30:13.339034+0000 32'63 77:136 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:35:34.462299+0000 4 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-09-04T19:30:13.684916+0000 32'63 77:154 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:29:24.461146+0000 1 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:13.442271+0000 32'63 77:140 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:21.795258+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745536+0000 0 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-09-04T19:30:13.977601+0000 32'64 77:150 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T04:40:09.694923+0000 2 0 2024-09-04T19:30:20.397 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:13.445530+0000 30'62 77:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745648+0000 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:13.439498+0000 32'63 77:126 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:11:05.599098+0000 2 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:30:13.976223+0000 32'64 77:144 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:21.839199+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovering+degraded 2024-09-04T19:30:13.488658+0000 69'65 77:150 [1,2,0] 1 [1,2,0] 1 37'66 2024-09-04T19:28:28.848408+0000 37'66 2024-09-04T19:28:25.765355+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:30:13.975746+0000 32'63 77:150 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:22.847326+0000 32'63 2024-09-04T19:28:22.847326+0000 0 1 periodic scrub scheduled @ 2024-09-05T23:12:28.799486+0000 4 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:13.683360+0000 32'63 77:138 [0,2,1] 0 [0,2,1] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovery_wait+degraded 2024-09-04T19:30:13.685335+0000 30'62 77:120 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:13.683723+0000 30'62 77:135 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:13.489310+0000 32'64 77:147 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:27.868947+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745786+0000 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:13.338841+0000 32'64 77:131 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745457+0000 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:13.917080+0000 32'63 77:143 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:23.855894+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:13.440158+0000 32'64 77:142 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:23.733961+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745475+0000 0 0 2024-09-04T19:30:20.398 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:13.445069+0000 30'62 77:135 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745619+0000 0 0 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:13.444044+0000 32'64 77:165 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745591+0000 0 0 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:13.488766+0000 30'62 77:117 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:22.938216+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745769+0000 0 0 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:13.441915+0000 32'63 77:131 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:21.743651+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 periodic scrub scheduled @ 2024-09-04T19:30:18.745511+0000 0 0 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:13.441404+0000 32'64 77:139 [1,0,2] 1 [1,0,2] 1 38'66 2024-09-04T19:28:23.763852+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:20.399 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-04T19:30:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:30:20.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:20.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:30:20.419 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:30:20.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:30:20.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:30:20.448 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:20.444+0000 7f8c840f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:20.450 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:20.448+0000 7f8c840f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:20.451 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:20.448+0000 7f8c840f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:20.849 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:20.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:20.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:20.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:30:21.259 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 74 up_thru 74 down_at 48 last_clean_interval [10,45) [v2:127.0.0.1:6818/3193162851,v1:127.0.0.1:6819/3193162851] [v2:127.0.0.1:6820/3193162851,v1:127.0.0.1:6821/3193162851] exists,up 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:30:21.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:30:21.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:21.261 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:21.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:21.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:21.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:30:21.444 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:21.440+0000 7f8c840f87c0 -1 Falling back to public interface 2024-09-04T19:30:21.703 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 74 up_thru 74 down_at 48 last_clean_interval [10,45) [v2:127.0.0.1:6818/3193162851,v1:127.0.0.1:6819/3193162851] [v2:127.0.0.1:6820/3193162851,v1:127.0.0.1:6821/3193162851] exists,up 2e03be93-3167-48bf-8313-79cddea604a4 2024-09-04T19:30:21.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:30:21.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:30:21.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:30:21.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-04T19:30:24.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:24.268+0000 7f8c840f87c0 -1 osd.1 77 log_to_monitors true 2024-09-04T19:30:41.707 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:41.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:30:41.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-04T19:30:41.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok dump_ops_in_flight 2024-09-04T19:30:41.826 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:30:41.827 INFO:tasks.workunit.client.0.smithi174.stdout: "ops": [], 2024-09-04T19:30:41.827 INFO:tasks.workunit.client.0.smithi174.stdout: "num_ops": 0 2024-09-04T19:30:41.827 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:30:41.840 INFO:tasks.workunit.client.0.smithi174.stdout:allowing recovery 2024-09-04T19:30:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-04T19:30:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:42.253 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:42.253 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:25.980229+0000 32'63 82:167 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:24.825823+0000 32'63 2024-09-04T19:28:24.825823+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:02:51.771919+0000 3 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.992446+0000 32'64 82:154 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:29.804962+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990350+0000 32'64 82:168 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:59:39.819458+0000 4 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:25.985992+0000 32'63 82:151 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:25.793935+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986868+0000 30'62 82:131 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:24.795794+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:25.988262+0000 32'64 82:143 [1,2,0] 1 [1,2,0] 1 37'65 2024-09-04T19:28:24.868034+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:30:25.989062+0000 32'64 82:173 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:41:13.965019+0000 1 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.983821+0000 30'62 82:151 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:26.855615+0000 30'62 2024-09-04T19:28:26.855615+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 5 0 0 0 0 3047 0 0 17 48 17 active+clean 2024-09-04T19:30:25.993100+0000 77'65 82:159 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:30.860452+0000 32'64 2024-09-04T19:28:30.860452+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-09-04T19:30:25.977832+0000 32'64 82:167 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:55:31.517183+0000 6 0 2024-09-04T19:30:42.254 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.989547+0000 30'62 82:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-09-04T19:30:25.990092+0000 32'63 82:144 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:43:52.593753+0000 4 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-09-04T19:30:25.980154+0000 32'63 82:168 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:27:03.267238+0000 1 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:25.985383+0000 32'63 82:148 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:21.795258+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-09-04T19:30:25.979469+0000 32'64 82:163 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:53:39.171082+0000 2 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986231+0000 30'62 82:142 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:25.993254+0000 32'63 82:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:55:40.173144+0000 2 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990658+0000 32'64 82:157 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:21.839199+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovery_wait+degraded 2024-09-04T19:30:25.990848+0000 69'65 82:156 [1,2,0] 1 [1,2,0] 1 37'66 2024-09-04T19:28:28.848408+0000 37'66 2024-09-04T19:28:25.765355+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:30:25.978584+0000 32'63 82:163 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:22.847326+0000 32'63 2024-09-04T19:28:22.847326+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:40:13.230972+0000 4 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.979025+0000 32'63 82:152 [0,2,1] 0 [0,2,1] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovering+degraded 2024-09-04T19:30:25.994984+0000 30'62 82:134 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:42.255 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.975173+0000 30'62 82:149 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.988929+0000 32'64 82:157 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:27.868947+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.994174+0000 32'64 82:139 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.981107+0000 32'63 82:156 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:23.855894+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.991206+0000 32'64 82:150 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:23.733961+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.987808+0000 30'62 82:143 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990140+0000 32'64 82:173 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986227+0000 30'62 82:126 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:22.938216+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.991226+0000 32'63 82:138 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:21.743651+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:25.986421+0000 32'64 82:146 [1,0,2] 1 [1,0,2] 1 38'66 2024-09-04T19:28:23.763852+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:42.256 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:42.257 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:42.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-04T19:30:42.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-04T19:30:42.428 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:30:42.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-04T19:30:42.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-04T19:30:42.597 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:30:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-04T19:30:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-04T19:30:42.760 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:30:42.775 INFO:tasks.workunit.client.0.smithi174.stdout:reading divergent objects 2024-09-04T19:30:42.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-04T19:30:42.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-04T19:30:43.179 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:30:43.179 INFO:tasks.workunit.client.0.smithi174.stdout:1.1f 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:25.980229+0000 32'63 82:167 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:24.825823+0000 32'63 2024-09-04T19:28:24.825823+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:02:51.771919+0000 3 0 2024-09-04T19:30:43.179 INFO:tasks.workunit.client.0.smithi174.stdout:1.1e 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.992446+0000 32'64 82:154 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:29.804962+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.1d 4 0 0 0 0 2468 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990350+0000 32'64 82:168 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:19.796680+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:59:39.819458+0000 4 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.b 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:25.985992+0000 32'63 82:151 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:25.793935+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.a 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986868+0000 30'62 82:131 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:24.795794+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.9 4 1 1 0 0 2430 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:25.988262+0000 32'64 82:143 [1,2,0] 1 [1,2,0] 1 37'65 2024-09-04T19:28:24.868034+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.8 3 0 0 0 0 1851 0 0 16 48 16 active+clean 2024-09-04T19:30:25.989062+0000 32'64 82:173 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:10.687391+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T00:41:13.965019+0000 1 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.7 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.983821+0000 30'62 82:151 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:26.855615+0000 30'62 2024-09-04T19:28:26.855615+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.6 5 0 0 0 0 3047 0 0 17 48 17 active+clean 2024-09-04T19:30:25.993100+0000 77'65 82:159 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:30.860452+0000 32'64 2024-09-04T19:28:30.860452+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.5 6 0 0 0 0 3702 0 0 16 48 16 active+clean 2024-09-04T19:30:25.977832+0000 32'64 82:167 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:18.874057+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T19:55:31.517183+0000 6 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.4 2 0 0 0 0 1196 0 0 14 48 14 active+clean 2024-09-04T19:30:25.989547+0000 30'62 82:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:11.699950+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 5 0 0 0 0 3085 0 0 15 48 15 active+clean 2024-09-04T19:30:25.990092+0000 32'63 82:144 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:08.582086+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T01:43:52.593753+0000 4 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 1 0 0 0 0 617 0 0 15 48 15 active+clean 2024-09-04T19:30:25.980154+0000 32'63 82:168 [0,1,2] 0 [0,1,2] 0 32'63 2024-09-04T19:28:19.796751+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T22:27:03.267238+0000 1 0 2024-09-04T19:30:43.180 INFO:tasks.workunit.client.0.smithi174.stdout:1.c 3 0 0 0 0 1813 0 0 15 48 15 active+clean 2024-09-04T19:30:25.985383+0000 32'63 82:148 [1,0,2] 1 [1,0,2] 1 32'63 2024-09-04T19:28:21.795258+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 2 0 0 0 0 1234 0 0 16 48 16 active+clean 2024-09-04T19:30:25.979469+0000 32'64 82:163 [2,0,1] 2 [2,0,1] 2 32'64 2024-09-04T19:28:16.846833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-05T21:53:39.171082+0000 2 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.1c 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986231+0000 30'62 82:142 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:19.779077+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 3 0 0 0 0 1851 0 0 15 48 15 active+clean 2024-09-04T19:30:25.993254+0000 32'63 82:134 [1,0,2] 1 [1,0,2] 1 30'62 2024-09-04T19:28:09.682092+0000 0'0 2024-09-04T19:27:43.895503+0000 0 1 periodic scrub scheduled @ 2024-09-06T05:55:40.173144+0000 2 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.d 4 0 0 0 0 2430 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990658+0000 32'64 82:157 [2,1,0] 2 [2,1,0] 2 32'64 2024-09-04T19:28:21.839199+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.e 4 1 1 0 0 2780 0 0 17 48 17 active+recovery_wait+degraded 2024-09-04T19:30:25.990848+0000 69'65 82:156 [1,2,0] 1 [1,2,0] 1 37'66 2024-09-04T19:28:28.848408+0000 37'66 2024-09-04T19:28:25.765355+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.f 4 0 0 0 0 2468 0 0 15 48 15 active+clean 2024-09-04T19:30:25.978584+0000 32'63 82:163 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:22.847326+0000 32'63 2024-09-04T19:28:22.847326+0000 0 1 periodic scrub scheduled @ 2024-09-06T06:40:13.230972+0000 4 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.10 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.979025+0000 32'63 82:152 [0,2,1] 0 [0,2,1] 0 32'63 2024-09-04T19:28:20.712663+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.11 2 0 1 0 0 1195 0 0 14 48 14 active+recovering+degraded 2024-09-04T19:30:25.994984+0000 30'62 82:134 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:15.945728+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.12 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.975173+0000 30'62 82:149 [0,1,2] 0 [0,1,2] 0 30'62 2024-09-04T19:28:20.870762+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.13 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.988929+0000 32'64 82:157 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:27.868947+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.14 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.994174+0000 32'64 82:139 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:15.727491+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.181 INFO:tasks.workunit.client.0.smithi174.stdout:1.15 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.981107+0000 32'63 82:156 [2,0,1] 2 [2,0,1] 2 32'63 2024-09-04T19:28:23.855894+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.16 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.991206+0000 32'64 82:150 [1,0,2] 1 [1,0,2] 1 32'64 2024-09-04T19:28:23.733961+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.17 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.987808+0000 30'62 82:143 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:12.716833+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.18 4 0 0 0 0 2429 0 0 16 48 16 active+clean 2024-09-04T19:30:25.990140+0000 32'64 82:173 [1,2,0] 1 [1,2,0] 1 32'64 2024-09-04T19:28:18.661030+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.19 2 0 0 0 0 1195 0 0 14 48 14 active+clean 2024-09-04T19:30:25.986227+0000 30'62 82:126 [1,2,0] 1 [1,2,0] 1 30'62 2024-09-04T19:28:22.938216+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.1a 3 0 0 0 0 1812 0 0 15 48 15 active+clean 2024-09-04T19:30:25.991226+0000 32'63 82:138 [1,2,0] 1 [1,2,0] 1 32'63 2024-09-04T19:28:21.743651+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 queued for scrub 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:1.1b 4 1 1 0 0 2429 0 0 16 48 16 active+recovery_wait+degraded 2024-09-04T19:30:25.986421+0000 32'64 82:146 [1,0,2] 1 [1,0,2] 1 38'66 2024-09-04T19:28:23.763852+0000 0'0 2024-09-04T19:27:43.895503+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:30:43.182 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:30:43.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-04T19:30:43.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-04T19:30:43.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:30:43.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-04T19:30:43.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:30:43.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-04T19:30:43.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:30:43.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-04T19:30:43.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:30:43.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-04T19:30:43.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-04T19:30:43.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:30:43.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-04T19:30:43.384 INFO:tasks.workunit.client.0.smithi174.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-04T19:30:43.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-04T19:30:43.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-04T19:30:43.419 INFO:tasks.workunit.client.0.smithi174.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-04T19:30:43.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-04T19:30:43.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-04T19:30:43.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'65 (29'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-09-04T19:28:21.185006+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 20 _merge_object_divergent_entries: keeping 37'65 (29'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-09-04T19:28:21.185006+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 37'65 last_divergent_update: 37'65 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: more recent entry found: 69'65 (29'4) modify 1:70b3bf12:::existing_4:head by client.4752.0:1 2024-09-04T19:29:26.103896+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'66 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-04T19:28:21.187039+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:30:43.442 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 20 _merge_object_divergent_entries: keeping 37'66 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-04T19:28:21.187039+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'66 last_divergent_update: 37'66 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.056+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 38'66 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-04T19:28:21.189571+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 20 _merge_object_divergent_entries: keeping 38'66 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-04T19:28:21.189571+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 38'66 last_divergent_update: 38'66 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-04T19:30:43.443 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 38'66 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 38'66 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-04T19:28:21.189571+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'48 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 38'65 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4625.0:1 2024-09-04T19:28:21.182543+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 20 _merge_object_divergent_entries: keeping 38'65 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4625.0:1 2024-09-04T19:28:21.182543+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 38'65 last_divergent_update: 38'65 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-04T19:30:43.444 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.132+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'65 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-09-04T19:28:21.190172+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 20 _merge_object_divergent_entries: keeping 37'65 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-09-04T19:28:21.190172+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 37'65 last_divergent_update: 37'65 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:30:43.445 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'65 2024-09-04T19:30:43.452 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'65 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-09-04T19:28:21.190172+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.136+0000 7f80a6385640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'48 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 20 _merge_object_divergent_entries: keeping 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 37'63 last_divergent_update: 37'63 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'63 2024-09-04T19:30:43.453 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:30:13.476+0000 7f80a7387640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'48 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 20 _merge_object_divergent_entries: keeping 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 37'63 last_divergent_update: 37'63 2024-09-04T19:30:43.454 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'63 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'63 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-04T19:28:21.185958+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.0.log:2024-09-04T19:30:13.136+0000 7fb01a562640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'48 2024-09-04T19:30:43.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-04T19:30:43.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:30:43.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-04T19:30:43.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:30:43.496 INFO:tasks.workunit.client.0.smithi174.stdout:success 2024-09-04T19:30:43.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-04T19:30:43.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27364.out 2024-09-04T19:30:43.497 INFO:tasks.workunit.client.0.smithi174.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27364.out': No such file or directory 2024-09-04T19:30:43.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-04T19:30:43.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-04T19:30:43.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-04T19:30:44.009 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' does not exist 2024-09-04T19:30:44.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-04T19:30:44.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:30:44.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:30:44.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:30:44.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:30:44.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:30:49.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:30:49.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-04T19:30:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:30:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:30:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:30:49.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:30:49.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:30:49.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:30:49.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:30:49.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:30:49.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:30:49.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:30:49.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:30:49.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:30:49.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:30:49.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:30:49.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:30:49.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:30:49.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:30:49.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:30:49.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:30:49.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:30:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:30:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:30:49.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:30:49.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:49.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:49.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:30:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:30:49.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:30:49.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:30:49.406 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:30:49.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:30:49.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:30:49.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:30:49.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:30:49.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:30:49.410 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:30:49.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:30:49.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:30:49.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:30:49.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:30:49.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:49.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:49.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:30:49.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:30:49.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:30:49.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-04T19:30:49.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:30:49.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:49.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:49.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27364 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-04T19:30:49.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-04T19:30:49.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-04T19:30:49.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-04T19:30:49.419 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-04T19:30:49.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:30:49.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-04T19:30:49.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-04T19:30:49.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:30:49.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:30:49.817 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:30:49.817 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:30:49.817 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:49.817 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:49.817 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:49.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:30:50.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:30:50.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:30:50.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:30:50.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:30:50.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:30:50.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:30:50.051 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:30:50.051 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:30:50.051 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:30:50.052 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:50.052 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:50.052 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:50.053 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:30:50.053 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:30:50.053 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get fsid 2024-09-04T19:30:50.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:30:50.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:30:50.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:30:50.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:30:50.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:30:50.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:30:50.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:30:50.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-mon.a.asok 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:30:50.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27364/ceph-mon.a.asok config get mon_host 2024-09-04T19:30:50.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-04T19:30:50.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-04T19:30:50.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:30:50.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:30:50.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:30:50.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-04T19:30:50.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:51.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:51.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:30:51.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:30:51.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:30:51.295 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:30:51.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:30:51.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:30:51.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:30:51.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:30:51.297 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:30:51.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:30:51.298 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:30:51.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 795d9795-7bb0-4876-a8c8-86c780524561' 2024-09-04T19:30:51.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:30:51.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrtdhme9ETExAAfszL1NiwiOpP5vUcoqhkug== 2024-09-04T19:30:51.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrtdhme9ETExAAfszL1NiwiOpP5vUcoqhkug=="}' 2024-09-04T19:30:51.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 795d9795-7bb0-4876-a8c8-86c780524561 -i td/divergent-priors/0/new.json 2024-09-04T19:30:51.616 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:51.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-04T19:30:51.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrtdhme9ETExAAfszL1NiwiOpP5vUcoqhkug== --osd-uuid 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:30:51.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:51.668+0000 7fe838d067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:51.683 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:51.680+0000 7fe838d067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:51.684 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:51.680+0000 7fe838d067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:51.684 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:51.680+0000 7fe838d067c0 -1 bdev(0x55ca2ddd6a80 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:30:51.684 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:51.680+0000 7fe838d067c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-04T19:30:58.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-04T19:30:58.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:30:58.057 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:30:58.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:30:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:30:58.640 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:30:58.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:30:58.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:30:58.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:30:58.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:30:58.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:30:58.680 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:58.676+0000 7f22827457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:58.713 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:58.708+0000 7f22827457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:58.715 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:58.712+0000 7f22827457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:30:59.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:30:59.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:30:59.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:30:59.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:30:59.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:30:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:30:59.757 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:30:59.752+0000 7f22827457c0 -1 Falling back to public interface 2024-09-04T19:31:00.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:00.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:00.526 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:31:00.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:31:00.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:00.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:01.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:01.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:01.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:31:01.943 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:31:01.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:01.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:02.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:02.598 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:02.596+0000 7f22827457c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:31:03.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:03.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:03.356 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:31:03.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:31:03.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:03.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:04.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:04.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:04.832 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:31:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:31:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:05.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:06.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:06.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:06.256 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:31:06.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:31:06.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:06.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:06.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:07.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:07.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:07.817 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:31:07.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:31:07.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:07.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:08.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:09.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:09.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:09.278 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:31:09.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:31:09.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:09.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:09.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:10.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:10.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:10.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:31:10.822 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:31:10.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:10.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:11.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:12.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:12.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:12.304 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:31:12.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:31:12.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:12.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:12.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:13.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:13.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:13.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:31:13.830 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:31:13.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:13.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:14.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:15.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:15.246 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:31:15.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:15.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:31:15.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:15.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:15.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:16.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:16.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:16.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:31:16.802 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:31:16.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:16.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:17.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:18.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:18.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:18.641 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:31:18.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:31:18.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:18.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:19.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:20.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:20.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:20.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:31:20.468 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:31:20.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:20.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:21.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:22.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:22.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:22.202 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:31:22.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:31:22.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:22.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:23.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:24.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:24.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:24.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:31:24.090 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:31:24.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:24.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:25.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:26.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:26.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:26.103 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:31:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:31:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:26.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:27.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:27.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:27.695 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:31:27.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:31:27.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:27.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:31:28.144 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3837392927,v1:127.0.0.1:6803/3837392927] [v2:127.0.0.1:6804/3837392927,v1:127.0.0.1:6805/3837392927] exists,up 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:31:28.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:31:28.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:31:28.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:31:28.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:31:28.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:31:28.151 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:31:28.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9b0711cd-b2f3-4c2b-8555-acb61edc4f50' 2024-09-04T19:31:28.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:31:28.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQtdhmR3dUChAA5ifb7u6/X15TZdgXqlO0lw== 2024-09-04T19:31:28.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQtdhmR3dUChAA5ifb7u6/X15TZdgXqlO0lw=="}' 2024-09-04T19:31:28.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 -i td/divergent-priors/1/new.json 2024-09-04T19:31:28.675 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:31:28.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-04T19:31:28.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQtdhmR3dUChAA5ifb7u6/X15TZdgXqlO0lw== --osd-uuid 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:31:28.737 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:28.731+0000 7fcc0fc317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:28.738 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:28.735+0000 7fcc0fc317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:28.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:28.735+0000 7fcc0fc317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:28.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:28.735+0000 7fcc0fc317c0 -1 bdev(0x55d0bbfc4a80 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:31:28.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:28.735+0000 7fcc0fc317c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-04T19:31:35.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-04T19:31:35.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:31:35.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:31:35.330 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:31:35.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:31:35.924 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:31:35.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:31:35.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:31:35.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:31:35.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:31:35.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:31:35.965 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:35.959+0000 7f2113ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:35.975 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:35.971+0000 7f2113ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:35.977 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:35.971+0000 7f2113ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:31:36.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:36.359 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:31:36.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:31:36.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:36.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:36.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:37.469 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:37.463+0000 7f2113ec57c0 -1 Falling back to public interface 2024-09-04T19:31:37.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:37.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:37.824 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:31:37.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:31:37.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:37.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:38.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:39.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:39.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:39.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:31:39.268 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:31:39.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:39.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:40.610 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:31:40.603+0000 7f2113ec57c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:31:40.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:40.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:40.723 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:31:40.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:31:40.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:40.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:41.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:42.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:42.354 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:31:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:31:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:42.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:43.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:43.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:43.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:31:43.869 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:31:43.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:43.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:44.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:45.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:45.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:45.298 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:31:45.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:31:45.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:45.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:45.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:46.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:46.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:46.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:31:46.799 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:31:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:47.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:48.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:48.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:48.242 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:31:48.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:31:48.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:48.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:48.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:49.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:49.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:49.908 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:31:49.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:31:49.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:49.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:50.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:51.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:51.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:51.455 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:31:51.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:31:51.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:51.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:51.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:52.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:52.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:52.878 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:31:52.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:31:52.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:52.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:53.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:54.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:54.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:54.473 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:31:54.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:31:54.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:54.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:55.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:56.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:56.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:56.513 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:31:56.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:31:56.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:56.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:57.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:31:58.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:31:58.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:31:58.115 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:31:58.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:31:58.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:31:58.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:31:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:00.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:00.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:00.387 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:32:00.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:32:00.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:00.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:32:01.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:02.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:02.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:02.247 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:32:02.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:32:02.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:02.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:32:03.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:04.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:04.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:04.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:32:04.043 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:32:04.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:04.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:32:04.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:05.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:05.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:05.574 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:32:05.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:32:05.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:05.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:32:06.002 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2227202624,v1:127.0.0.1:6811/2227202624] [v2:127.0.0.1:6812/2227202624,v1:127.0.0.1:6813/2227202624] exists,up 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:32:06.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:32:06.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:32:06.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:32:06.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-04T19:32:06.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:32:06.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:32:06.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:32:06.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:32:06.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:32:06.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:32:06.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 269d155a-affe-470f-a22d-92a11487f32b' 2024-09-04T19:32:06.010 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:32:06.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:32:06.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2tdhmqCXPARAAOnDqPhWHUDhws6PIU1c7WA== 2024-09-04T19:32:06.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2tdhmqCXPARAAOnDqPhWHUDhws6PIU1c7WA=="}' 2024-09-04T19:32:06.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 269d155a-affe-470f-a22d-92a11487f32b -i td/divergent-priors/2/new.json 2024-09-04T19:32:06.521 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:32:06.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-04T19:32:06.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2tdhmqCXPARAAOnDqPhWHUDhws6PIU1c7WA== --osd-uuid 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:32:06.580 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:06.579+0000 7fb0cac9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:06.609 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:06.607+0000 7fb0cac9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:06.611 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:06.611+0000 7fb0cac9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:06.611 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:06.611+0000 7fb0cac9b7c0 -1 bdev(0x562923b70a80 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:32:06.611 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:06.611+0000 7fb0cac9b7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-04T19:32:13.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-04T19:32:13.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:32:13.281 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:32:13.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:32:13.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:32:13.863 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:32:13.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:32:13.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:32:13.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:32:13.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:32:13.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:32:13.904 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:13.903+0000 7f0fa6b657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:13.937 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:13.935+0000 7f0fa6b657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:13.939 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:13.939+0000 7f0fa6b657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:32:14.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:32:14.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:14.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:32:14.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:14.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:14.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:14.989 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:14.987+0000 7f0fa6b657c0 -1 Falling back to public interface 2024-09-04T19:32:15.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:15.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:15.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:32:15.811 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:32:15.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:15.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:16.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:17.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:32:17.296 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:32:17.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:17.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:17.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:17.920 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:32:17.919+0000 7f0fa6b657c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:32:18.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:18.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:18.735 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:32:18.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:32:18.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:18.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:19.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:20.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:20.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:20.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:32:20.196 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:32:20.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:20.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:20.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:21.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:21.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:21.672 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:32:21.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:32:21.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:21.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:22.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:23.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:23.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:23.198 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:32:23.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:32:23.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:23.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:23.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:24.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:24.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:24.651 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:32:24.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:32:24.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:24.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:25.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:26.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:26.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:26.118 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:32:26.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:32:26.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:26.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:26.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:27.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:27.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:27.634 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:32:27.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:32:27.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:27.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:28.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:29.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:29.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:29.213 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:32:29.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:32:29.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:29.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:29.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:30.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:30.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:30.657 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:32:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:32:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:31.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:32.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:32.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:32.307 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:32:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:32:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:33.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:34.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:34.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:34.290 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:32:34.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:32:34.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:34.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:35.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:36.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:36.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:36.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:32:36.475 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:32:36.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:36.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:37.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:38.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:38.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:38.272 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:32:38.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:32:38.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:38.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:39.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:40.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:40.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:40.481 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:32:40.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:32:40.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:40.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:41.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:42.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:42.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:42.213 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:32:42.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:32:42.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:42.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:42.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:32:43.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:32:43.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:32:43.925 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:32:43.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:32:43.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:32:43.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:32:44.419 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/596686859,v1:127.0.0.1:6819/596686859] [v2:127.0.0.1:6820/596686859,v1:127.0.0.1:6821/596686859] exists,up 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:32:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:32:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:32:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:32:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-04T19:32:45.133 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-04T19:32:45.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-04T19:32:46.154 INFO:tasks.workunit.client.0.smithi174.stderr:noin is set 2024-09-04T19:32:46.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-04T19:32:47.067 INFO:tasks.workunit.client.0.smithi174.stderr:nodown is set 2024-09-04T19:32:47.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-04T19:32:47.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=test 2024-09-04T19:32:47.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-09-04T19:32:47.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-09-04T19:32:47.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-09-04T19:32:47.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-04T19:32:47.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-04T19:32:47.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-04T19:32:48.707 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' already exists 2024-09-04T19:32:48.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:32:49.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-09-04T19:32:49.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-04T19:32:50.907 INFO:tasks.workunit.client.0.smithi174.stderr:set pool 1 allow_ec_overwrites to true 2024-09-04T19:32:50.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-09-04T19:32:50.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:32:50.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:32:50.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:32:50.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:32:50.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:32:50.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:32:50.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:32:50.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:32:50.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:32:51.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:32:51.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:32:51.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:32:51.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:32:51.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:32:51.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:51.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:32:51.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-09-04T19:32:51.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-09-04T19:32:51.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-09-04T19:32:51.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:51.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:32:51.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-09-04T19:32:51.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-09-04T19:32:51.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-09-04T19:32:51.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:51.779 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-64424509450' 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-09-04T19:32:51.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:51.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:32:51.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-09-04T19:32:51.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:51.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-09-04T19:32:51.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-04T19:32:51.959 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836502 2024-09-04T19:32:51.959 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:32:52.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-04T19:32:52.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:52.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-09-04T19:32:52.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:52.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:32:52.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-09-04T19:32:52.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:52.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-09-04T19:32:52.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-04T19:32:52.382 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672973 2024-09-04T19:32:52.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:32:52.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-04T19:32:52.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:52.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-09-04T19:32:52.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:52.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:32:52.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-09-04T19:32:52.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:52.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-09-04T19:32:52.829 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509450 2024-09-04T19:32:52.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-04T19:32:52.829 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:32:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509450 2024-09-04T19:32:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:32:54.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:32:54.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:32:54.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-04T19:32:54.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:32:54.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:32:54.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:32:55.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:32:55.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:32:55.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:32:55.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:32:55.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:32:55.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:32:55.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:32:56.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:56.615 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:32:56.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-09-04T19:32:56.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-09-04T19:32:56.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-09-04T19:32:56.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:56.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:32:56.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-09-04T19:32:56.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-09-04T19:32:56.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672976' 2024-09-04T19:32:56.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:32:56.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:32:57.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-09-04T19:32:57.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-09-04T19:32:57.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672976 2-64424509453' 2024-09-04T19:32:57.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:57.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:57.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-09-04T19:32:57.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:32:57.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-09-04T19:32:57.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:57.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-09-04T19:32:57.101 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836505 2024-09-04T19:32:57.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-04T19:32:57.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:32:57.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-04T19:32:57.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:32:58.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:32:58.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:32:58.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-04T19:32:58.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:58.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-09-04T19:32:58.940 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:58.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:32:58.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-09-04T19:32:58.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:58.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-09-04T19:32:58.944 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672976 2024-09-04T19:32:58.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-04T19:32:58.944 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:32:59.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-04T19:32:59.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:32:59.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-09-04T19:32:59.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:32:59.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:32:59.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-09-04T19:32:59.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:32:59.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-09-04T19:32:59.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-04T19:32:59.372 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509453 2024-09-04T19:32:59.372 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:32:59.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:32:59.802 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:32:59.802 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:32:59.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:32:59.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:32:59.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:32:59.912 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:33:00.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:33:00.341 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:33:00.342 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:33:00.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:33:00.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:00.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:33:00.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-09-04T19:33:00.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-09-04T19:33:00.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-09-04T19:33:00.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:00.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:33:00.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-09-04T19:33:00.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-09-04T19:33:00.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672979' 2024-09-04T19:33:00.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:00.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:33:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-09-04T19:33:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-09-04T19:33:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672979 2-64424509456' 2024-09-04T19:33:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:00.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-09-04T19:33:00.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:00.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:33:00.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-09-04T19:33:00.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:00.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-09-04T19:33:00.819 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836508 2024-09-04T19:33:00.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-04T19:33:00.819 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:33:01.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-04T19:33:01.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:33:02.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:33:02.225 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:33:02.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-04T19:33:02.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:02.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-09-04T19:33:02.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:02.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:33:02.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-09-04T19:33:02.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:02.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-09-04T19:33:02.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-04T19:33:02.662 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672979 2024-09-04T19:33:02.662 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:33:03.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-04T19:33:03.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:03.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-09-04T19:33:03.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:03.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:33:03.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-09-04T19:33:03.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:03.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-09-04T19:33:03.109 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509456 2024-09-04T19:33:03.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-04T19:33:03.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:33:03.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-04T19:33:03.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:33:03.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:33:03.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:33:04.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:33:04.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:33:04.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:33:04.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:33:04.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:33:04.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:33:04.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:33:05.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:33:05.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:33:05.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:33:05.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-04T19:33:05.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-04T19:33:05.424 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:33:05.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-04T19:33:05.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-04T19:33:05.441 INFO:tasks.workunit.client.0.smithi174.stdout:primary and soon to be divergent is 1 2024-09-04T19:33:05.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:33:05.811 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:33:05.811 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-04T19:32:50.198441+0000 0'0 29:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T06:29:14.870062+0000 0 0 2024-09-04T19:33:05.811 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:33:05.811 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:33:05.811 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-04T19:33:05.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:05.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-04T19:33:05.827 INFO:tasks.workunit.client.0.smithi174.stdout:writing initial objects 2024-09-04T19:33:05.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-04T19:33:05.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-04T19:33:05.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-04T19:33:05.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:05.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-04T19:33:06.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:06.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-04T19:33:06.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:06.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-04T19:33:06.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:06.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-04T19:33:06.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:06.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-04T19:33:06.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:06.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-04T19:33:07.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:07.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-04T19:33:07.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:07.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-04T19:33:07.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:07.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-04T19:33:07.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:07.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-04T19:33:08.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:08.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-04T19:33:08.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:08.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-04T19:33:08.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:08.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-04T19:33:08.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:08.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-04T19:33:09.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:09.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-04T19:33:09.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:09.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-04T19:33:09.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:09.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-04T19:33:09.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:09.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-04T19:33:09.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:09.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-04T19:33:10.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:10.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-04T19:33:10.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:10.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-04T19:33:10.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:10.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-04T19:33:10.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:10.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-04T19:33:11.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:11.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-04T19:33:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-04T19:33:11.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:11.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-04T19:33:11.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:11.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-04T19:33:12.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:12.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-04T19:33:12.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:12.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-04T19:33:12.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:12.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-04T19:33:12.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:12.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-04T19:33:12.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:12.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-04T19:33:13.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:13.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-04T19:33:13.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:13.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-04T19:33:13.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:13.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-04T19:33:13.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:13.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-04T19:33:14.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:14.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-04T19:33:14.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:14.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-04T19:33:14.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:14.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-04T19:33:14.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:14.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-04T19:33:15.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:15.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-04T19:33:15.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:15.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-04T19:33:15.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:15.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-04T19:33:15.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:15.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-04T19:33:15.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:15.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-04T19:33:16.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:16.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-04T19:33:16.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:16.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-04T19:33:16.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:16.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-04T19:33:16.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:16.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-04T19:33:17.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:17.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-04T19:33:17.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:17.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-04T19:33:17.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:17.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-04T19:33:17.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:17.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-04T19:33:18.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:18.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-04T19:33:18.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:18.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-04T19:33:18.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:18.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-04T19:33:18.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:18.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-04T19:33:19.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:19.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-04T19:33:19.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:19.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-04T19:33:19.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:19.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-04T19:33:19.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:19.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-04T19:33:19.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:19.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-04T19:33:20.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:20.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-04T19:33:20.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:20.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-04T19:33:20.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:20.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-04T19:33:20.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:20.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-04T19:33:21.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:21.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-04T19:33:21.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:21.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-04T19:33:21.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:21.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-04T19:33:21.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:21.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-04T19:33:22.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:22.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-04T19:33:22.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:22.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-04T19:33:22.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:22.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-04T19:33:22.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:22.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-04T19:33:23.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:23.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-04T19:33:23.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:23.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-04T19:33:23.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:23.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-04T19:33:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-04T19:33:24.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:24.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-04T19:33:24.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:24.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-04T19:33:24.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:24.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-04T19:33:24.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:24.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-04T19:33:24.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:24.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-04T19:33:25.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:25.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-04T19:33:25.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:25.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-04T19:33:25.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:25.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-04T19:33:25.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:25.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-04T19:33:26.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:26.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-04T19:33:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-04T19:33:26.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:26.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-04T19:33:26.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:26.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-04T19:33:27.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:27.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-04T19:33:27.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:27.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-04T19:33:27.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:27.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-04T19:33:27.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:27.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-04T19:33:27.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:27.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-04T19:33:28.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-04T19:33:28.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:28.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-04T19:33:28.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:28.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-04T19:33:28.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-04T19:33:28.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:33:29.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:33:29.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:33:29.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:33:29.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:33:29.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:33:29.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:33:29.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:33:29.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:33:29.725 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:33:29.726 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:33:29.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:33:29.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:29.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:33:29.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-09-04T19:33:29.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-09-04T19:33:29.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-09-04T19:33:29.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:29.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:33:30.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-09-04T19:33:30.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-09-04T19:33:30.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672986' 2024-09-04T19:33:30.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:33:30.064 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:33:30.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-09-04T19:33:30.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-09-04T19:33:30.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509464' 2024-09-04T19:33:30.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:30.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-09-04T19:33:30.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:30.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:33:30.225 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-09-04T19:33:30.225 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:30.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-09-04T19:33:30.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-04T19:33:30.227 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836516 2024-09-04T19:33:30.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:33:30.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-09-04T19:33:30.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:33:31.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:33:31.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-04T19:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-09-04T19:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:32.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:33:32.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-09-04T19:33:32.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:32.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-09-04T19:33:32.077 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672986 2024-09-04T19:33:32.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-04T19:33:32.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:33:32.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-09-04T19:33:32.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:33:32.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-09-04T19:33:32.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:33:32.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:33:32.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:33:32.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-09-04T19:33:32.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-09-04T19:33:32.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-04T19:33:32.571 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509464 2024-09-04T19:33:32.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:33:33.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-04T19:33:33.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:33:33.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:33:33.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:33:33.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:33:33.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:33:33.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:33:33.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:33:33.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:33:33.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:33:33.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:33:33.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:33:34.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:33:34.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:33:34.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:33:34.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:33:34.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:33:34.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:33:34.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:33:34.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-04T19:33:34.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-04T19:33:34.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-04T19:33:34.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-04T19:33:34.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-04T19:33:35.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-04T19:33:35.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-04T19:33:35.066 INFO:tasks.workunit.client.0.smithi174.stdout:blackholing osds 0 2 2024-09-04T19:33:35.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:33:35.480 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:33:35.480 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-04T19:32:50.198441+0000 29'100 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T06:29:14.870062+0000 0 0 2024-09-04T19:33:35.480 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:33:35.480 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:33:35.480 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:33:35.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:35.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-04T19:33:35.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:33:35.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:33:35.497 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:33:35.497 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:33:35.497 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:33:35.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:33:35.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-04T19:33:35.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-04T19:33:35.605 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:33:35.605 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:33:35.605 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:33:35.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:35.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-04T19:33:35.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:33:35.619 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:33:35.619 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:33:35.619 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:33:35.619 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:33:35.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:33:35.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-04T19:33:35.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-04T19:33:35.725 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:33:35.725 INFO:tasks.workunit.client.0.smithi174.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-04T19:33:35.725 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:33:35.736 INFO:tasks.workunit.client.0.smithi174.stdout:writing divergent object 2024-09-04T19:33:35.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-04T19:33:35.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-04T19:33:35.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-04T19:33:36.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-04T19:33:36.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-04T19:33:36.964 INFO:tasks.workunit.client.0.smithi174.stdout:created pool test snap snap1 2024-09-04T19:33:36.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-04T19:33:36.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-04T19:33:36.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-04T19:33:46.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-04T19:33:46.976 INFO:tasks.workunit.client.0.smithi174.stderr:rados: no process found 2024-09-04T19:33:46.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-04T19:33:46.977 INFO:tasks.workunit.client.0.smithi174.stdout:killing all the osds 2024-09-04T19:33:46.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:33:47.393 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:33:47.393 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-09-04T19:32:50.198441+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:52:55.599026+0000 0 0 2024-09-04T19:33:47.393 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:33:47.393 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:33:47.394 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:33:47.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-04T19:33:47.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:33:47.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:33:47.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:33:47.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:33:47.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:33:47.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:33:47.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:47.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-04T19:33:48.045 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:33:48.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:48.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-04T19:33:48.647 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:33:48.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-04T19:33:48.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-04T19:33:49.215 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:33:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-04T19:33:49.774 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-04T19:33:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-04T19:33:50.392 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-04T19:33:50.408 INFO:tasks.workunit.client.0.smithi174.stdout:bringing up non_divergent 0 2 2024-09-04T19:33:50.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-04T19:33:50.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:33:50.808 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:33:50.808 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-09-04T19:32:50.198441+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-05T20:52:55.599026+0000 0 0 2024-09-04T19:33:50.808 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:33:50.808 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:33:50.808 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:33:50.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:33:50.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:33:50.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:33:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:33:50.831 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:33:50.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:33:50.832 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-04T19:33:50.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:33:50.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:33:50.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:33:50.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:33:50.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:50.861+0000 7f1be53e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:50.867 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:50.865+0000 7f1be53e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:50.868 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:50.865+0000 7f1be53e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:51.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:33:51.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:33:51.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:33:51.242 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:33:51.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:33:51.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:33:51.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:51.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:33:51.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:33:51.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:51.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:33:52.097 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:52.093+0000 7f1be53e17c0 -1 Falling back to public interface 2024-09-04T19:33:52.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:33:52.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:52.680 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:33:52.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:33:52.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:52.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:33:53.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:33:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:33:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:54.168 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:33:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:33:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:33:54.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:33:55.367 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:55.365+0000 7f1be53e17c0 -1 osd.0 29 log_to_monitors true 2024-09-04T19:33:55.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:33:55.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:55.620 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:33:55.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:33:55.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:55.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:33:56.199 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3557843333,v1:127.0.0.1:6803/3557843333] [v2:127.0.0.1:6804/3557843333,v1:127.0.0.1:6805/3557843333] exists,up 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:33:56.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:33:56.204 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:33:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:33:56.206 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:33:56.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:33:56.243 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:56.241+0000 7fc0045277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:56.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:56.269+0000 7fc0045277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:56.273 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:56.269+0000 7fc0045277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:56.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:33:56.668 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:33:56.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:56.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:33:57.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:33:57.537 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:33:57.533+0000 7fc0045277c0 -1 Falling back to public interface 2024-09-04T19:33:58.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:33:58.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:58.102 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:33:58.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:33:58.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:58.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:33:58.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:33:59.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:33:59.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:33:59.542 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:33:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:33:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:33:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:33:59.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:34:00.981 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:00.977+0000 7fc0045277c0 -1 osd.2 29 log_to_monitors true 2024-09-04T19:34:00.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:34:00.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:34:00.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:34:00.993 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:34:00.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:34:00.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:34:01.587 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/705420118,v1:127.0.0.1:6811/705420118] [v2:127.0.0.1:6812/705420118,v1:127.0.0.1:6813/705420118] exists,up 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:34:01.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:34:01.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:34:01.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:34:01.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:34:01.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-04T19:34:02.141 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already in. 2024-09-04T19:34:02.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:34:02.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-04T19:34:03.074 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-04T19:34:03.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-04T19:34:08.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-04T19:34:08.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-04T19:34:08.100 INFO:tasks.workunit.client.0.smithi174.stdout:writing non-divergent object existing_4 2024-09-04T19:34:08.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-04T19:34:08.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:34:08.510 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:34:08.510 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-04T19:34:03.282477+0000 29'100 43:127 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T06:25:45.074037+0000 0 0 2024-09-04T19:34:08.510 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:34:08.510 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:34:08.510 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:34:08.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-04T19:34:08.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:34:08.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-04T19:34:08.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:34:08.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:34:08.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:34:08.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:34:08.639 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:34:08.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:34:08.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:34:08.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:34:08.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:34:08.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:34:08.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:34:08.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:34:08.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:34:08.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:34:08.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:34:09.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:34:09.140 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:34:09.140 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:34:09.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:34:09.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:09.140 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:34:09.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789957 2024-09-04T19:34:09.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789957 2024-09-04T19:34:09.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789957' 2024-09-04T19:34:09.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:09.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:34:09.431 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:34:09.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:34:09.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:34:09.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:34:09.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:09.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:34:09.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724548 2024-09-04T19:34:09.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724548 2024-09-04T19:34:09.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-04T19:34:09.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:34:09.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-158913789957 2024-09-04T19:34:09.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:34:09.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:34:09.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-158913789957 2024-09-04T19:34:09.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:34:09.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789957 2024-09-04T19:34:09.591 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 158913789957 2024-09-04T19:34:09.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-04T19:34:09.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:34:10.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-04T19:34:10.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:34:10.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:34:10.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724548 2024-09-04T19:34:10.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:34:10.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724548 2024-09-04T19:34:10.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:34:10.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724548 2024-09-04T19:34:10.015 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 167503724548 2024-09-04T19:34:10.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-04T19:34:10.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:34:10.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724548 2024-09-04T19:34:10.444 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:34:10.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:10.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:11.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:34:11.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:11.014 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:11.014 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:11.014 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:11.014 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:11.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:11.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:11.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:11.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:11.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:11.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:11.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:11.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:34:11.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:34:11.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:34:11.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:11.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:34:12.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:12.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:12.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:12.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:12.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:12.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:12.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:12.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:12.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:12.513 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:12.513 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:12.513 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:13.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:13.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:13.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:13.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:13.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:34:13.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:13.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:34:13.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:13.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:13.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:13.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:13.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:13.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:13.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:13.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:14.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:14.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:14.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:14.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:14.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:15.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:15.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:15.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:34:15.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:15.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:34:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:15.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:15.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:15.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:15.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:15.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:15.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:16.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:16.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:16.231 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:16.231 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:16.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:16.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:16.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:16.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:16.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:16.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:16.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:16.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:16.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:34:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:17.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:34:18.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:18.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:18.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:18.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:18.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:18.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:18.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:18.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:18.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:19.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:19.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:19.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:19.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:19.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:34:19.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:19.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:34:21.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:21.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:21.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:21.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:21.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:21.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:21.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:21.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:21.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:21.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:21.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:21.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:22.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:22.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:22.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:22.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:22.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:22.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:34:22.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:22.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:34:25.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:25.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:25.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:25.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:25.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:25.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:25.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:25.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:26.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:26.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:26.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:26.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:26.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:34:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:34:33.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:33.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:33.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:33.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:33.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:33.878 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:33.878 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:33.878 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:34.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:34.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:34.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:34.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:34.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:34.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:35.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:35.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:35.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:34:35.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:35.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:34:42.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:42.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:42.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:42.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:42.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:42.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:42.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:42.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:43.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:43.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:43.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:43.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:43.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:43.682 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:44.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:44.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:44.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:34:44.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:34:44.808 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:34:44.808 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "188e919c-590c-4aba-8108-ff14d5916e47", 2024-09-04T19:34:44.808 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:34:44.807068+0000", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.809 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.810 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:34:44.811 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.812 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:34:44.813 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:30:49.452190Z", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:30:49.452190Z", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:34:44.814 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.815 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.816 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 43, 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:34:44.817 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:30:50.387130+0000", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:34:02.846021+0000", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:34:01.283965+0000", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:34:02.681969+0000", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:34:44.818 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 10, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.819 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:32:48.393491+0000", 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 16389, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 3, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 1, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:34:44.820 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "off", 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 1, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 1, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 1, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 1, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 1, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:34:44.821 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "30", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "pool", 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 1, 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 30, 2024-09-04T19:34:44.822 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [ 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "snapid": 1, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "stamp": "2024-09-04T19:33:36.784136+0000", 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "snap1" 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:34:44.823 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "myprofile", 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:34:44.824 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 8192, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {} 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:34:44.825 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "795d9795-7bb0-4876-a8c8-86c780524561", 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 5, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 37, 2024-09-04T19:34:44.826 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 42, 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 31, 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.827 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.828 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.829 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.830 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3557843333 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.831 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/3557843333", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/3557843333", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3557843333", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3557843333", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "9b0711cd-b2f3-4c2b-8555-acb61edc4f50", 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:34:44.832 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 25, 2024-09-04T19:34:44.833 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 32, 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.834 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.835 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:34:44.836 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:34:44.837 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/2227202624", 2024-09-04T19:34:44.838 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/2227202624", 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2227202624", 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2227202624", 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "269d155a-affe-470f-a22d-92a11487f32b", 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:34:44.839 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 15, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 29, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 39, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 0, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 33, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.840 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:34:44.841 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:34:44.842 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.843 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.844 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.845 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.846 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 705420118 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.847 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/705420118", 2024-09-04T19:34:44.848 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/705420118", 2024-09-04T19:34:44.848 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/705420118", 2024-09-04T19:34:44.848 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/705420118", 2024-09-04T19:34:44.848 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:34:44.848 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:34:44.849 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.850 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:34:44.850 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:33:47.878960+0000", 2024-09-04T19:34:44.850 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:34:44.850 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:34:44.850 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:31:26.256010+0000", 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:33:48.438174+0000", 2024-09-04T19:34:44.852 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:34:44.852 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:34:44.852 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:34:44.852 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:34:44.852 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:32:04.266946+0000", 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:33:49.039114+0000", 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:34:44.853 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:32:42.935879+0000", 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.854 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:34:44.855 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "myprofile": { 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-device-class": "", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-failure-domain": "osd", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-num-failure-domains": "0", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-osds-per-failure-domain": "0", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-root": "default", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:34:44.856 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "1", 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van", 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "w": "8" 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:34:44.857 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/3557843333,v1:127.0.0.1:6805/3557843333]", 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:34:44.858 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:34:44.859 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:30:57.369588Z", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:34:44.860 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/3557843333,v1:127.0.0.1:6803/3557843333]", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3557843333,v1:127.0.0.1:6809/3557843333]", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3557843333,v1:127.0.0.1:6807/3557843333]", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:34:44.861 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/2227202624,v1:127.0.0.1:6813/2227202624]", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:34:44.862 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:34:44.863 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:31:34.599084Z", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/2227202624,v1:127.0.0.1:6811/2227202624]", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2227202624,v1:127.0.0.1:6817/2227202624]", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2227202624,v1:127.0.0.1:6815/2227202624]", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:34:44.864 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:34:44.865 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/705420118,v1:127.0.0.1:6813/705420118]", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:34:44.866 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:32:12.585334Z", 2024-09-04T19:34:44.867 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/705420118,v1:127.0.0.1:6811/705420118]", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/705420118,v1:127.0.0.1:6817/705420118]", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/705420118,v1:127.0.0.1:6815/705420118]", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:34:44.868 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.869 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 0, 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [] 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 37 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:34:44.870 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 39 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 43, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:34:44.871 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.872 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.873 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:34:44.874 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:34:44.875 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:34:44.876 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:34:44.877 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:34:44.878 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.879 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.880 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.881 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:34:44.882 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.883 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:34:44.884 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.885 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:34:44.886 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.887 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 1, 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "test", 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 3, 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "set_chooseleaf_tries", 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 5 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.888 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "set_choose_tries", 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 100 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:34:44.889 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_indep", 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:34:44.890 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stderr:report 1807990195 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:34:44.891 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:34:44.892 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:30:50:354470+0000", 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:34:44.893 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:34:44.894 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:34:44.895 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 1, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62667, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 1, 2024-09-04T19:34:44.896 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 303, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:34:44.897 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:34:44.898 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:34:44.899 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 18, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 18, 2024-09-04T19:34:44.900 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:34:44.901 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83696, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2164, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489104, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036842496, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2215936, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1550717, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:34:44.902 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.903 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:34:44.904 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 83696, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2164, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 10, 2024-09-04T19:34:44.905 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 81461, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314489104, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322036842496, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2215936, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1550717, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 10396, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 83416932 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:34:44.906 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.907 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62667, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 1, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 303, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:34:44.908 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:34:44.909 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:34:44.910 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1241088, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1241088, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:34:44.911 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 18, 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 18, 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 3, 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 3, 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:34:44.912 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 158913789964 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672990 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 167503724555 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:34:44.913 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 1 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2147483647, 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:34:44.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 1, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.915 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:34:44.916 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 5, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:33:45.846922+0000", 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 106, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 417, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 571, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:34:44.917 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:34:44.918 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:34:44.918 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:34:44.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:34:44.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:34:44.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:34:44.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-04T19:34:44.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-04T19:34:46.560 INFO:tasks.workunit.client.0.smithi174.stderr:PG '1.0' not found 2024-09-04T19:34:46.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-04T19:34:46.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-04T19:34:46.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:34:46.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:34:46.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:34:46.873 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:34:46.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:34:46.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-04T19:34:46.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:34:46.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:34:46.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:34:46.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:34:46.915 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:46.913+0000 7f4944cd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:46.917 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:46.913+0000 7f4944cd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:46.919 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:46.917+0000 7f4944cd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:47.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:34:47.319 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:34:47.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:34:47.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:34:47.750 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3557843333,v1:127.0.0.1:6803/3557843333] [v2:127.0.0.1:6804/3557843333,v1:127.0.0.1:6805/3557843333] exists,up 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:34:47.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:34:47.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:34:47.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:34:47.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:34:47.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-04T19:34:47.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:34:47.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:34:47.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:34:47.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:34:47.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:34:47.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:34:47.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-04T19:34:47.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-04T19:34:48.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:48.129+0000 7f4944cd67c0 -1 Falling back to public interface 2024-09-04T19:34:49.428 INFO:tasks.workunit.client.0.smithi174.stderr:PG '1.0' not found 2024-09-04T19:34:49.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-04T19:34:49.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-04T19:34:49.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:34:49.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:34:49.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:34:49.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:34:49.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:34:49.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-04T19:34:49.734 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:34:49.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:34:49.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:34:49.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-04T19:34:49.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:34:49.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:34:49.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:34:49.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:34:49.778 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:49.777+0000 7f4fabc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:49.780 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:49.777+0000 7f4fabc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:49.782 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:49.777+0000 7f4fabc527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:34:50.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:34:50.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:34:50.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:34:50.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:34:50.770 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/705420118,v1:127.0.0.1:6811/705420118] [v2:127.0.0.1:6812/705420118,v1:127.0.0.1:6813/705420118] exists,up 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-04T19:34:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-04T19:34:51.081 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:51.077+0000 7f4944cd67c0 -1 osd.0 43 log_to_monitors true 2024-09-04T19:34:51.292 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:51.289+0000 7f4fabc527c0 -1 Falling back to public interface 2024-09-04T19:34:52.996 INFO:tasks.workunit.client.0.smithi174.stderr:PG '1.0' not found 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:34:53.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:34:53.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-04T19:34:53.568 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:34:53.568 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:34:53.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:34:53.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:34:53.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:34:53.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-04T19:34:53.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:34:53.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:34:53.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:34:53.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:34:53.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:34:53.930 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:34:53.925+0000 7f4fabc527c0 -1 osd.2 43 log_to_monitors true 2024-09-04T19:34:54.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:34:54.105 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:34:54.105 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-04T19:34:54.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:34:54.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:54.105 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:34:54.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495619 2024-09-04T19:34:54.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495619 2024-09-04T19:34:54.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495619' 2024-09-04T19:34:54.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:54.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:34:54.505 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-04T19:34:54.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:34:54.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:34:54.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:34:54.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:34:54.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:34:54.664 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-04T19:34:54.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-04T19:34:54.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-04T19:34:54.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-04T19:34:54.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:34:54.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495619 2024-09-04T19:34:54.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:34:54.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:34:54.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495619 2024-09-04T19:34:54.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:34:54.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495619 2024-09-04T19:34:54.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-09-04T19:34:54.670 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 197568495619 2024-09-04T19:34:54.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:34:55.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 197568495619 2024-09-04T19:34:55.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:34:56.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:34:56.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:34:56.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-04T19:34:56.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:34:56.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:56.537 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:57.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:34:57.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:57.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:57.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:57.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:57.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:57.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:57.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:57.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:57.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:57.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:57.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:58.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-04T19:34:58.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:58.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:58.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:58.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:58.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:58.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:58.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:58.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:34:58.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:34:58.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:34:58.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:34:58.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:34:59.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:34:59.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:34:59.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:34:59.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:34:59.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:34:59.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-04T19:34:59.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:34:59.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-04T19:34:59.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:34:59.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:34:59.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:34:59.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:34:59.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:34:59.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:34:59.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:34:59.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:00.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:00.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:00.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:00.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:00.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:00.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:00.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:01.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:01.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:01.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-04T19:35:01.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:01.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-04T19:35:01.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:01.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:01.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:02.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:02.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:02.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:02.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:02.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:02.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:03.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:03.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:03.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-04T19:35:03.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:03.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:04.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:04.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:04.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:04.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:04.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:04.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:04.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:05.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:05.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:05.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-04T19:35:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-04T19:35:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:07.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:07.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:07.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:07.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:07.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:07.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:07.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:07.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:07.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:07.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:08.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:08.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:08.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:08.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:08.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:08.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-04T19:35:08.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:08.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-04T19:35:12.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:12.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:12.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:12.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:12.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:12.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:12.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:12.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:12.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:12.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:12.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:12.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:13.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:13.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:13.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:13.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:13.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:13.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-04T19:35:13.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:13.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-04T19:35:19.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:19.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:19.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:19.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:19.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:19.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:19.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:19.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:20.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:20.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:20.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:20.381 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:20.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:20.911 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:20.911 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:21.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-04T19:35:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-04T19:35:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-04T19:35:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-04T19:35:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:35:28.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:35:28.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:35:28.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:35:28.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:35:28.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:35:28.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:35:29.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-04T19:35:29.238 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:35:29.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:35:29.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-04T19:35:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-04T19:35:29.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-04T19:35:29.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-04T19:35:30.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-04T19:35:30.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-04T19:35:30.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-04T19:35:30.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_fingerprint": "188e919c-590c-4aba-8108-ff14d5916e47", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "19.1.1-235-g1ef86450", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "commit": "1ef864504b8875c83ee6c2c5fedc13315bebf7f5", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "timestamp": "2024-09-04T19:35:30.891481+0000", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "tag": "", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "health": { 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "HEALTH_WARN", 2024-09-04T19:35:30.893 INFO:tasks.workunit.client.0.smithi174.stdout: "checks": { 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "OSDMAP_FLAGS": { 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 17 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [], 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "OSD_DOWN": { 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:35:30.894 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 osds down", 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "osd.1 (root=default,host=smithi174) is down" 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "PG_DEGRADED": { 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:35:30.895 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "severity": "HEALTH_WARN", 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "summary": { 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-04T19:35:30.896 INFO:tasks.workunit.client.0.smithi174.stdout: "count": 1 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: "detail": [ 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "application not enabled on pool 'test'" 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: "muted": false 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.897 INFO:tasks.workunit.client.0.smithi174.stdout: "mutes": [] 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_first_committed": 1, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap_last_committed": 1, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "monmap": { 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:30:49.452190Z", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:30:49.452190Z", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release": 19, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "min_mon_release_name": "squid", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "election_strategy": 1, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "disallowed_leaders: ": "", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": false, 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "tiebreaker_mon": "", 2024-09-04T19:35:30.898 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_ranks: ": "", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "features": { 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "persistent": [ 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "kraken", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "luminous", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "mimic", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap-prune", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "nautilus", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "octopus", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "pacific", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "elector-pinging", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "quincy", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "reef", 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "squid" 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.899 INFO:tasks.workunit.client.0.smithi174.stdout: "optional": [] 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "mons": [ 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "rank": 0, 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "a", 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115", 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 0 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.900 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:7115/0", 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "priority": 0, 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 0, 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_location": "{}" 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "quorum": [ 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: 0 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap": { 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 52, 2024-09-04T19:35:30.901 INFO:tasks.workunit.client.0.smithi174.stdout: "fsid": "0693bb67-36af-43e9-9a6e-b4da686af1ad", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "created": "2024-09-04T19:30:50.387130+0000", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:34:55.228245+0000", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "last_up_change": "2024-09-04T19:34:54.525518+0000", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "last_in_change": "2024-09-04T19:34:02.681969+0000", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_num": 5800384, 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_set": [ 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "nodown", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "noin", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "noout", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "pglog_hardlimit", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snapdirs", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_deletes", 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: "sortbitwise" 2024-09-04T19:35:30.902 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_version": 12, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "full_ratio": 0.99000000953674316, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_snapshot": "", 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_max": 1, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "max_osd": 3, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "require_min_compat_client": "luminous", 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "min_compat_client": "jewel", 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "require_osd_release": "squid", 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "allow_crimson": false, 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "pools": [ 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.903 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_name": "test", 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "create_time": "2024-09-04T19:32:48.393491+0000", 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": 16389, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 3, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "size": 3, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "min_size": 2, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_rule": 1, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_count": 0, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_target": 0, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_barrier": 0, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "object_hash": 2, 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_autoscale_mode": "off", 2024-09-04T19:35:30.904 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num": 1, 2024-09-04T19:35:30.905 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num": 1, 2024-09-04T19:35:30.905 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_placement_num_target": 1, 2024-09-04T19:35:30.905 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_target": 1, 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_num_pending": 1, 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pg_merge_meta": { 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "source_pgid": "0.0", 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "ready_epoch": 0, 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 0, 2024-09-04T19:35:30.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 0, 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "source_version": "0'0", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "target_version": "0'0" 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "30", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend": "0", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "last_force_op_resend_preluminous": "0", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "auid": 0, 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_mode": "pool", 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_seq": 1, 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_epoch": 30, 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_snaps": [ 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snapid": 1, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "stamp": "2024-09-04T19:33:36.784136+0000", 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "snap1" 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps": "[]", 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_bytes": 0, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "quota_max_objects": 0, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "tiers": [], 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "tier_of": -1, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "read_tier": -1, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "write_tier": -1, 2024-09-04T19:35:30.912 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_mode": "none", 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_bytes": 0, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "target_max_objects": 0, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_flush_age": 0, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "cache_min_evict_age": 0, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profile": "myprofile", 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_params": { 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "none" 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.913 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_period": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_count": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "use_gmt_hitset": true, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "min_read_recency_for_promote": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "min_write_recency_for_promote": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_grade_decay_rate": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_search_last_n": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "grade_table": [], 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "stripe_width": 8192, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "expected_num_objects": 0, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "fast_read": false, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "options": {}, 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: "application_metadata": {} 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.914 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "osds": [ 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "795d9795-7bb0-4876-a8c8-86c780524561", 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 37, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 43, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 46, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 51, 2024-09-04T19:35:30.915 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 45, 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6802", 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6803", 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.916 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6804", 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6805", 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.917 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6808", 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6809", 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.918 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6806", 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6807", 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 3686479956 2024-09-04T19:35:30.919 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6803/3686479956", 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6805/3686479956", 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3686479956", 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3686479956", 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.920 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "9b0711cd-b2f3-4c2b-8555-acb61edc4f50", 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 0, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 0, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 0, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 10, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 25, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 32, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.921 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.922 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.923 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.924 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2227202624 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/2227202624", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/2227202624", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2227202624", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2227202624", 2024-09-04T19:35:30.925 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "exists" 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "uuid": "269d155a-affe-470f-a22d-92a11487f32b", 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 1, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "in": 1, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 1, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_affinity": 1, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_begin": 39, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_end": 43, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 50, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "up_thru": 0, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "down_at": 49, 2024-09-04T19:35:30.926 INFO:tasks.workunit.client.0.smithi174.stdout: "lost_at": 0, 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addrs": { 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6810", 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6811", 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.927 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addrs": { 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6812", 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6813", 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addrs": { 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.928 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6816", 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6817", 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addrs": { 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "addrvec": [ 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.929 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v2", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6814", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "v1", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "addr": "127.0.0.1:6815", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "nonce": 2557806959 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "public_addr": "127.0.0.1:6811/2557806959", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "cluster_addr": "127.0.0.1:6813/2557806959", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2557806959", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2557806959", 2024-09-04T19:35:30.930 INFO:tasks.workunit.client.0.smithi174.stdout: "state": [ 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "exists", 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "up" 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_xinfo": [ 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:34:51.431771+0000", 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:35:30.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:31:26.256010+0000", 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:33:48.438174+0000", 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:32:04.266946+0000", 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.932 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "down_stamp": "2024-09-04T19:34:54.316334+0000", 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_probability": 0, 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "laggy_interval": 0, 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "features": 4540701547738038271, 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "old_weight": 0, 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "last_purged_snaps_scrub": "2024-09-04T19:32:42.935879+0000", 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "dead_epoch": 0 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap": [], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_items": [], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_upmap_primaries": [], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_temp": [], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "primary_temp": [], 2024-09-04T19:35:30.933 INFO:tasks.workunit.client.0.smithi174.stdout: "blocklist": {}, 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "range_blocklist": {}, 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "erasure_code_profiles": { 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "default": { 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "2", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van" 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "myprofile": { 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-device-class": "", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-failure-domain": "osd", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-num-failure-domains": "0", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-osds-per-failure-domain": "0", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "crush-root": "default", 2024-09-04T19:35:30.934 INFO:tasks.workunit.client.0.smithi174.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "k": "2", 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "m": "1", 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "plugin": "jerasure", 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "technique": "reed_sol_van", 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "w": "8" 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "removed_snaps_queue": [], 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "new_removed_snaps": [], 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "new_purged_snaps": [], 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "crush_node_flags": {}, 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "device_class_flags": {}, 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode": { 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_enabled": false, 2024-09-04T19:35:30.935 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_bucket_count": 0, 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "degraded_stretch_mode": 0, 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering_stretch_mode": 0, 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "stretch_mode_bucket": 0 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_metadata": [ 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6804/3686479956,v1:127.0.0.1:6805/3686479956]", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:35:30.936 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:35:30.937 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:30:57.369588Z", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6802/3686479956,v1:127.0.0.1:6803/3686479956]", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3686479956,v1:127.0.0.1:6809/3686479956]", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3686479956,v1:127.0.0.1:6807/3686479956]", 2024-09-04T19:35:30.938 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/0", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:35:30.939 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/2227202624,v1:127.0.0.1:6813/2227202624]", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:35:30.940 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:31:34.599084Z", 2024-09-04T19:35:30.941 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/2227202624,v1:127.0.0.1:6811/2227202624]", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2227202624,v1:127.0.0.1:6817/2227202624]", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2227202624,v1:127.0.0.1:6815/2227202624]", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:35:30.942 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/1", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "arch": "x86_64", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "back_addr": "[v2:127.0.0.1:6812/2557806959,v1:127.0.0.1:6813/2557806959]", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "back_iface": "", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs": "1", 2024-09-04T19:35:30.943 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_db": "0", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_dedicated_wal": "0", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluefs_single_shared_device": "1", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_allocation_from_file": "0", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_access_mode": "file", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_block_size": "4096", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_devices": "sda", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_rotational": "1", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_size": "107374182400", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_support_discard": "0", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_bdev_type": "hdd", 2024-09-04T19:35:30.944 INFO:tasks.workunit.client.0.smithi174.stdout: "bluestore_min_alloc_size": "4096", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_release": "squid", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_short": "19.1.1-235-g1ef86450", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "ceph_version_when_created": "ceph version 19.1.1-235-g1ef86450 (1ef864504b8875c83ee6c2c5fedc13315bebf7f5) squid (rc)", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "created_at": "2024-09-04T19:32:12.585334Z", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "default_device_class": "hdd", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XN3Z", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": "sda", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "distro": "ubuntu", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "distro_version": "22.04", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "front_addr": "[v2:127.0.0.1:6810/2557806959,v1:127.0.0.1:6811/2557806959]", 2024-09-04T19:35:30.945 INFO:tasks.workunit.client.0.smithi174.stdout: "front_iface": "", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2557806959,v1:127.0.0.1:6817/2557806959]", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2557806959,v1:127.0.0.1:6815/2557806959]", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "hostname": "smithi174", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "journal_rotational": "1", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_swap_kb": "2097148", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "mem_total_kb": "32746748", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "os": "Linux", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_data": "td/divergent-priors/2", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_objectstore": "bluestore", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "osdspec_affinity": "", 2024-09-04T19:35:30.946 INFO:tasks.workunit.client.0.smithi174.stdout: "rotational": "1" 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_clean_epochs": { 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "min_last_epoch_clean": 10, 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": { 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "per_pool": [ 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "floor": 26 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_epochs": [ 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:35:30.947 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 46 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 10 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 50 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_first_committed": 1, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "osdmap_last_committed": 52, 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "crushmap": { 2024-09-04T19:35:30.948 INFO:tasks.workunit.client.0.smithi174.stdout: "devices": [ 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.0", 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.1", 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd.2", 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: "class": "hdd" 2024-09-04T19:35:30.949 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "types": [ 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 0, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "osd" 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "host" 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 2, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "chassis" 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.950 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 3, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "rack" 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 4, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "row" 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 5, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pdu" 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 6, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "pod" 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.951 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 7, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "room" 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 8, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "datacenter" 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 9, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "zone" 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 10, 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "region" 2024-09-04T19:35:30.952 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "root" 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "buckets": [ 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -1, 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default", 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:35:30.953 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -2, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "default~hdd", 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 11, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "root", 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:35:30.954 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -3, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174", 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:35:30.955 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.956 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "id": -4, 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "smithi174~hdd", 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "type_id": 1, 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "type_name": "host", 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 19206, 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "alg": "straw2", 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": "rjenkins1", 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "items": [ 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 0, 2024-09-04T19:35:30.957 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 0 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 1, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 1 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "id": 2, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "weight": 6402, 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "pos": 2 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.958 INFO:tasks.workunit.client.0.smithi174.stdout: "rules": [ 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 0, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "replicated_rule", 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 1, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_firstn", 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.959 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_id": 1, 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "rule_name": "test", 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "type": 3, 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "steps": [ 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "set_chooseleaf_tries", 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 5 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.960 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "set_choose_tries", 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 100 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "take", 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "item": -1, 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "item_name": "default" 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "choose_indep", 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 0, 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "type": "osd" 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.961 INFO:tasks.workunit.client.0.smithi174.stdout: "op": "emit" 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "tunables": { 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_tries": 0, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_local_fallback_tries": 0, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_total_tries": 50, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_descend_once": 1, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_vary_r": 1, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "chooseleaf_stable": 1, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_descents": 100, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "msr_collision_tries": 100, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "straw_calc_version": 1, 2024-09-04T19:35:30.962 INFO:tasks.workunit.client.0.smithi174.stdout: "allowed_bucket_algs": 54, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "profile": "jewel", 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "optimal_tunables": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "legacy_tunables": 0, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "minimum_required_version": "jewel", 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables2": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v2_rules": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables3": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v3_rules": 0, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v4_buckets": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "require_feature_tunables5": 1, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "has_v5_rules": 0, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "has_msr_rules": 0 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: "choose_args": {} 2024-09-04T19:35:30.963 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "fsmap": { 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 1, 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "btime": "2024-09-04T19:30:50:354470+0000", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "default_fscid": -1, 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": { 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "compat": {}, 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "ro_compat": {}, 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "incompat": { 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_1": "base v0.20", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_2": "client writeable ranges", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_3": "default file layouts on dirs", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_4": "dir inode in separate object", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_5": "mds uses versioned encoding", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-04T19:35:30.964 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_8": "no anchor table", 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_9": "file layout v2", 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_10": "snaprealm v2", 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_11": "minor log segments", 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_12": "quiesce subvolumes" 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "feature_flags": { 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "enable_multiple": true, 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "ever_enabled_multiple": true 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "standbys": [], 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "filesystems": [] 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_first_committed": 1, 2024-09-04T19:35:30.965 INFO:tasks.workunit.client.0.smithi174.stdout: "mdsmap_last_committed": 1, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "auth": { 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 5, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_secrets": 3 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_active": 1, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_unknown": 0, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osd": 3, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_sum": { 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62667, 2024-09-04T19:35:30.966 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 1, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 303, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:35:30.967 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:35:30.968 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 0, 2024-09-04T19:35:30.969 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 0, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 18, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 18, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 3, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 3, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 0 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum": { 2024-09-04T19:35:30.970 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 85140, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2308, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 15, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 82736, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314487660, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322035363840, 2024-09-04T19:35:30.971 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2363392, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1611901, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 16263, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 84721785 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:35:30.972 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_sum_by_class": { 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "hdd": { 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "up_from": 0, 2024-09-04T19:35:30.973 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 0, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pgs": 3, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "num_osds": 3, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_osds": 3, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "num_per_pool_omap_osds": 3, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb": 314572800, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used": 85140, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_data": 2308, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_omap": 15, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_used_meta": 82736, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "kb_avail": 314487660, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "statfs": { 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 322122547200, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 322035363840, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:35:30.974 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 2363392, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1611901, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 16263, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 84721785 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "hb_peers": [], 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trim_queue_len": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "num_snap_trimming": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shards_repaired": 0, 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "op_queue_age_hist": { 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "histogram": [], 2024-09-04T19:35:30.975 INFO:tasks.workunit.client.0.smithi174.stdout: "upper_bound": 1 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "perf_stat": { 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ms": 0, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ms": 0, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "commit_latency_ns": 0, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "apply_latency_ns": 0 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "alerts": [], 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "network_ping_times": [] 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "pool_stats": [ 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "poolid": 1, 2024-09-04T19:35:30.976 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg": 1, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 62667, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 1, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 303, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-04T19:35:30.977 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 101, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-04T19:35:30.978 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "store_stats": { 2024-09-04T19:35:30.979 INFO:tasks.workunit.client.0.smithi174.stdout: "total": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "available": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "internally_reserved": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "allocated": 1241088, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "data_stored": 1241088, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_allocated": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "data_compressed_original": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_allocated": 0, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "internal_metadata": 0 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 18, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 18, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "up": 3, 2024-09-04T19:35:30.980 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": 3, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "num_store_stats": 3 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "osd_stats": [ 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 197568495626 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 42949672990 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:35:30.981 INFO:tasks.workunit.client.0.smithi174.stdout: "seq": 214748364808 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_state": [ 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num": 1 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num_pg_by_osd": [ 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2147483647, 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:35:30.982 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 2, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 0, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 1, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 1, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.983 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": 1, 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "num_primary_pg": 0, 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "num_acting_pg": 0, 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "num_up_not_acting_pg": 0 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [ 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 1, 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": {} 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "servicemap": { 2024-09-04T19:35:30.984 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 5, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "modified": "2024-09-04T19:33:45.846922+0000", 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "services": {} 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_first_committed": 1, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "mgrstat_last_committed": 129, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_first_committed": 1, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "logm_last_committed": 499, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "paxos": { 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "first_committed": 1, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "last_committed": 684, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "last_pn": 0, 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: "accepted_pn": 0 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-04T19:35:30.985 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:35:30.986 INFO:tasks.workunit.client.0.smithi174.stderr:report 3559781926 2024-09-04T19:35:30.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-04T19:35:30.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-04T19:35:30.986 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery 2024-09-04T19:35:30.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:31.324 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:31.324 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-04T19:34:55.539470+0000 43'102 52:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:35:31.324 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:31.324 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:31.324 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:31.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:35:31.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-04T19:35:31.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-04T19:35:31.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-04T19:35:31.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:35:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:35:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:35:31.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.0.asok 2024-09-04T19:35:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-04T19:35:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.0.asok set_recovery_delay 100000 2024-09-04T19:35:31.452 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:35:31.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-04T19:35:31.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-04T19:35:31.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-04T19:35:31.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-04T19:35:31.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:35:31.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:35:31.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:35:31.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.2.asok 2024-09-04T19:35:31.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-04T19:35:31.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.2.asok set_recovery_delay 100000 2024-09-04T19:35:31.582 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:35:31.596 INFO:tasks.workunit.client.0.smithi174.stdout:revive divergent 1 2024-09-04T19:35:31.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-04T19:35:31.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:31.982 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:31.983 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-04T19:34:55.539470+0000 43'102 52:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:35:31.983 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:31.983 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:31.983 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:31.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-04T19:35:32.690 INFO:tasks.workunit.client.0.smithi174.stderr:noup is set 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:35:32.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:35:32.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:35:32.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:35:32.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:35:32.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:35:32.717 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:35:32.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:35:32.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:35:32.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:35:32.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:35:32.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:35:32.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:35:32.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:35:32.766 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:32.764+0000 7fe19409b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:32.773 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:32.768+0000 7fe19409b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:32.775 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:32.772+0000 7fe19409b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-04T19:35:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-04T19:35:33.789 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:33.784+0000 7fe19409b7c0 -1 Falling back to public interface 2024-09-04T19:35:36.590 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:36.588+0000 7fe19409b7c0 -1 osd.1 30 log_to_monitors true 2024-09-04T19:35:38.130 INFO:tasks.workunit.client.0.smithi174.stdout:delay recovery divergent 2024-09-04T19:35:38.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-04T19:35:38.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:38.542 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:38.542 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-04T19:34:55.539470+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T04:57:54.656393+0000 0 0 2024-09-04T19:35:38.542 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:38.542 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:38.542 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:38.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-04T19:35:38.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-04T19:35:38.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27364/ceph-osd.1.asok 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-04T19:35:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27364/ceph-osd.1.asok set_recovery_delay 100000 2024-09-04T19:35:38.669 INFO:tasks.workunit.client.0.smithi174.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-04T19:35:38.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-04T19:35:39.333 INFO:tasks.workunit.client.0.smithi174.stderr:noup is unset 2024-09-04T19:35:39.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:35:39.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:35:39.811 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 46 up_thru 51 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/3686479956,v1:127.0.0.1:6803/3686479956] [v2:127.0.0.1:6804/3686479956,v1:127.0.0.1:6805/3686479956] exists,up 795d9795-7bb0-4876-a8c8-86c780524561 2024-09-04T19:35:39.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:35:39.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:35:39.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:35:39.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:35:39.813 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:35:39.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:35:39.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:35:40.306 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1465576030,v1:127.0.0.1:6819/1465576030] [v2:127.0.0.1:6820/1465576030,v1:127.0.0.1:6821/1465576030] exists,up 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:35:40.307 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:35:40.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:35:40.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:35:40.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:35:40.706 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/2557806959,v1:127.0.0.1:6811/2557806959] [v2:127.0.0.1:6812/2557806959,v1:127.0.0.1:6813/2557806959] exists,up 269d155a-affe-470f-a22d-92a11487f32b 2024-09-04T19:35:40.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:35:40.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:35:40.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:35:40.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:41.118 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:41.119 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-04T19:34:55.539470+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T04:57:54.656393+0000 0 0 2024-09-04T19:35:41.119 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:41.119 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:41.119 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:41.136 INFO:tasks.workunit.client.0.smithi174.stdout:wait for peering 2024-09-04T19:35:41.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-04T19:35:41.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:41.517 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:41.517 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-04T19:34:55.539470+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T04:57:54.656393+0000 0 0 2024-09-04T19:35:41.517 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:41.517 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:41.517 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:41.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-04T19:35:41.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-04T19:35:41.644 INFO:tasks.workunit.client.0.smithi174.stdout:killing divergent 1 2024-09-04T19:35:41.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:42.028 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:42.028 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 102 4 4 0 0 63284 0 0 13 90 13 active+recovering+degraded 2024-09-04T19:35:40.283695+0000 58'103 58:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:35:42.028 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:42.028 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:42.028 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:42.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-04T19:35:42.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:35:42.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:35:42.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:35:42.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:35:42.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:35:42.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:35:42.154 INFO:tasks.workunit.client.0.smithi174.stdout:reviving divergent 1 2024-09-04T19:35:42.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-04T19:35:42.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:35:42.530 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:35:42.530 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 102 4 4 0 0 63284 0 0 13 90 13 active+recovering+degraded 2024-09-04T19:35:40.283695+0000 58'103 58:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 no scrub is scheduled 0 0 2024-09-04T19:35:42.530 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:35:42.530 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:35:42.530 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:35:42.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-04T19:35:42.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-04T19:35:42.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-04T19:35:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' 2024-09-04T19:35:42.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:35:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-04T19:35:42.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:35:42.552 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0693bb67-36af-43e9-9a6e-b4da686af1ad --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27364/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:35:42.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:35:42.589 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:42.584+0000 7f831f0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:42.591 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:42.588+0000 7f831f0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:42.593 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:42.588+0000 7f831f0c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:35:42.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:35:42.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:35:42.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:35:42.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:35:42.951 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:35:42.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:35:42.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:35:42.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:35:42.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:35:42.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:35:43.404 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1465576030,v1:127.0.0.1:6819/1465576030] [v2:127.0.0.1:6820/1465576030,v1:127.0.0.1:6821/1465576030] exists,up 9b0711cd-b2f3-4c2b-8555-acb61edc4f50 2024-09-04T19:35:43.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:35:43.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:35:43.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:35:43.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-04T19:35:43.809 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:43.804+0000 7f831f0c17c0 -1 Falling back to public interface 2024-09-04T19:35:47.226 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:35:47.220+0000 7f831f0c17c0 -1 osd.1 58 log_to_monitors true 2024-09-04T19:36:03.406 INFO:tasks.workunit.client.0.smithi174.stdout:allowing recovery 2024-09-04T19:36:03.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-04T19:36:03.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:36:03.807 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:36:03.808 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-04T19:35:49.244793+0000 58'103 63:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T04:58:47.308632+0000 0 0 2024-09-04T19:36:03.808 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:36:03.808 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:36:03.808 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:36:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-04T19:36:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-04T19:36:03.990 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:36:04.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-04T19:36:04.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-04T19:36:04.154 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:36:04.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-04T19:36:04.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-04T19:36:04.319 INFO:tasks.workunit.client.0.smithi174.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-04T19:36:04.333 INFO:tasks.workunit.client.0.smithi174.stdout:reading divergent objects 2024-09-04T19:36:04.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-04T19:36:04.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-04T19:36:04.775 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-04T19:36:04.775 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-04T19:35:49.244793+0000 58'103 63:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-04T19:32:48.448072+0000 0'0 2024-09-04T19:32:48.448072+0000 0 0 periodic scrub scheduled @ 2024-09-06T04:58:47.308632+0000 0 0 2024-09-04T19:36:04.775 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-04T19:36:04.775 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-04T19:36:04.776 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-04T19:36:04.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-04T19:36:04.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-04T19:36:04.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:36:04.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-04T19:36:04.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:36:04.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-04T19:36:04.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:36:04.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-04T19:36:04.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-04T19:36:04.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-04T19:36:04.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-04T19:36:04.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-04T19:36:04.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:36:04.968 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-04T19:33:28.691321+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:36:04.968 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-04T19:33:28.691321+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-04T19:36:04.968 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:36:04.968 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-04T19:36:04.969 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-04T19:36:04.969 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-04T19:36:04.969 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-09-04T19:33:37.013108+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:36:04.969 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-09-04T19:33:37.013108+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:36:04.969 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:36:04.970 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-04T19:36:04.970 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-04T19:36:04.970 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:36:04.970 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-04T19:36:04.970 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-09-04T19:33:37.013108+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-04T19:36:04.971 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-04T19:36:04.971 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-04T19:33:35.776564+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4658.0:1 2024-09-04T19:33:36.784710+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:36:04.972 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-04T19:33:35.776564+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:36:04.972 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4658.0:1 2024-09-04T19:33:36.784710+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:36:04.972 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-04T19:36:04.972 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-04T19:36:04.973 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-04T19:36:04.973 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-04T19:36:04.973 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-04T19:36:04.973 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4658.0:1 2024-09-04T19:33:36.784710+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-04T19:36:04.973 INFO:tasks.workunit.client.0.smithi174.stdout:td/divergent-priors/osd.1.log:2024-09-04T19:35:40.188+0000 7fe178025640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-04T19:36:04.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-04T19:36:04.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-04T19:36:04.977 INFO:tasks.workunit.client.0.smithi174.stdout:success 2024-09-04T19:36:04.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-04T19:36:04.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-04T19:36:04.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-04T19:36:04.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-04T19:36:05.474 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'test' does not exist 2024-09-04T19:36:05.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-04T19:36:05.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:36:05.493 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:36:05.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:36:05.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:36:05.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:36:05.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:36:05.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-04T19:36:05.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:36:05.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:36:05.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:36:05.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:36:05.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:36:05.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:36:05.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:36:05.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:36:05.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:36:05.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:36:05.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:36:05.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:36:05.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:36:05.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:36:05.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:36:05.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:36:05.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:36:05.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:36:05.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:36:05.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:36:05.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:36:05.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:05.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:36:05.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:36:05.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:36:05.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/divergent-priors 0 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:36:05.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:36:05.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:36:05.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:36:05.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:36:05.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:36:05.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:36:05.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:36:05.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:36:05.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:36:05.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:36:05.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:36:05.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-04T19:36:05.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-04T19:36:05.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27364 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27364 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:36:05.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-04T19:36:05.852 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-04T19:36:05.852 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-04T19:36:05.905 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-04T19:36:05.905 DEBUG:teuthology.orchestra.run.smithi174:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-04T19:36:05.956 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-04T19:36:05.960 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-04T19:36:05.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ec-error-rollforward 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-04T19:36:05.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-04T19:36:05.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-04T19:36:05.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none ' 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-04T19:36:05.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:36:05.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:36:05.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:36:05.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:36:05.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:36:05.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:36:05.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:36:05.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:36:05.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:36:05.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:36:05.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:36:05.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:36:05.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:36:05.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:36:05.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:36:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:36:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-04T19:36:05.973 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:36:05.973 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:05.973 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:05.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84609 2024-09-04T19:36:05.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:36:05.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:36:05.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-09-04T19:36:05.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:36:05.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:05.975 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:05.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.84609 2024-09-04T19:36:05.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:36:05.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-09-04T19:36:05.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:06.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:06.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:36:06.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:36:06.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:36:06.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:36:06.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:36:06.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:36:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:06.635 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:06.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84609/ceph-mon.a.asok 2024-09-04T19:36:06.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:36:06.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.84609/ceph-mon.a.asok config get fsid 2024-09-04T19:36:06.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:36:06.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:36:06.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:36:06.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:36:06.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:36:06.747 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:36:06.747 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84609/ceph-mon.a.asok 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:36:06.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.84609/ceph-mon.a.asok config get mon_host 2024-09-04T19:36:06.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-04T19:36:06.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-09-04T19:36:06.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:36:06.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:36:06.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:36:06.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-09-04T19:36:06.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:07.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:07.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:36:07.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:36:07.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:07.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-04T19:36:07.784 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:36:07.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f6a11dd-e321-4df6-969d-fb779dde24e5 2024-09-04T19:36:07.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2f6a11dd-e321-4df6-969d-fb779dde24e5' 2024-09-04T19:36:07.785 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 2f6a11dd-e321-4df6-969d-fb779dde24e5 2024-09-04T19:36:07.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:36:07.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCntthmguOzLxAAq4HtbgOJH51GPVHPbrGGBA== 2024-09-04T19:36:07.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCntthmguOzLxAAq4HtbgOJH51GPVHPbrGGBA=="}' 2024-09-04T19:36:07.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f6a11dd-e321-4df6-969d-fb779dde24e5 -i td/ec-error-rollforward/0/new.json 2024-09-04T19:36:08.084 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:36:08.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-04T19:36:08.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCntthmguOzLxAAq4HtbgOJH51GPVHPbrGGBA== --osd-uuid 2f6a11dd-e321-4df6-969d-fb779dde24e5 2024-09-04T19:36:08.131 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:08.128+0000 7fa2abf887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:08.153 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:08.148+0000 7fa2abf887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:08.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:08.148+0000 7fa2abf887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:08.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:08.148+0000 7fa2abf887c0 -1 bdev(0x55f0bb6b2a80 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:36:08.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:08.152+0000 7fa2abf887c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-04T19:36:14.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-04T19:36:14.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:36:14.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:36:14.516 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:36:14.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:36:15.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:36:15.050 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:36:15.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:36:15.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:36:15.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:36:15.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:36:15.079 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:15.076+0000 7fd73753d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:15.082 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:15.076+0000 7fd73753d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:15.083 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:15.080+0000 7fd73753d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:36:15.446 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:36:15.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:15.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:36:15.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:15.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:15.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:16.593 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:16.588+0000 7fd73753d7c0 -1 Falling back to public interface 2024-09-04T19:36:16.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:16.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:16.933 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:36:16.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:36:16.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:16.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:17.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:18.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:18.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:18.331 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:36:18.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:36:18.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:18.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:18.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:19.441 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:19.436+0000 7fd73753d7c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:36:19.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:19.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:19.756 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:36:19.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:36:19.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:19.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:20.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:21.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:21.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:21.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:36:21.200 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:36:21.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:21.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:21.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:22.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:22.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:22.665 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:36:22.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:36:22.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:22.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:23.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:24.071 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:24.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:25.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:25.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:25.485 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:36:25.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:36:25.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:25.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:25.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:26.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:26.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:26.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:36:26.933 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:36:26.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:26.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:27.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:28.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:28.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:28.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:36:28.441 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:36:28.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:28.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:28.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:29.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:29.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:29.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:36:29.909 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:36:29.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:29.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:30.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:31.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:31.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:31.393 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:36:31.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:36:31.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:31.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:31.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:32.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:32.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:32.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:36:32.855 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:36:32.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:32.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:33.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:34.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:34.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:34.400 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:36:34.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:36:34.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:34.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:34.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:35.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:35.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:35.968 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:36:35.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:36:35.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:35.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:36.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:37.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:37.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:37.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:36:37.802 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:36:37.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:37.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:38.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:39.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:39.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:39.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:36:39.552 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:36:39.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:39.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:40.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:41.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:41.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:41.296 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:36:41.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:36:41.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:41.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:42.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:43.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:43.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:43.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:36:43.396 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:36:43.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:43.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:43.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:44.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:44.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:44.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-04T19:36:44.874 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-04T19:36:44.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:44.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:36:45.275 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/50983507,v1:127.0.0.1:6803/50983507] [v2:127.0.0.1:6804/50983507,v1:127.0.0.1:6805/50983507] exists,up 2f6a11dd-e321-4df6-969d-fb779dde24e5 2024-09-04T19:36:45.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:36:45.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:36:45.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-04T19:36:45.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:36:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-04T19:36:45.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:36:45.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba86f7ae-7246-49a9-8ce8-4a54c46519f9 2024-09-04T19:36:45.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ba86f7ae-7246-49a9-8ce8-4a54c46519f9' 2024-09-04T19:36:45.283 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 ba86f7ae-7246-49a9-8ce8-4a54c46519f9 2024-09-04T19:36:45.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:36:45.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNtthmWCWuERAAo1DjNJGdVLsHKyq52Sy5cg== 2024-09-04T19:36:45.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNtthmWCWuERAAo1DjNJGdVLsHKyq52Sy5cg=="}' 2024-09-04T19:36:45.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba86f7ae-7246-49a9-8ce8-4a54c46519f9 -i td/ec-error-rollforward/1/new.json 2024-09-04T19:36:45.779 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:36:45.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-04T19:36:45.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNtthmWCWuERAAo1DjNJGdVLsHKyq52Sy5cg== --osd-uuid ba86f7ae-7246-49a9-8ce8-4a54c46519f9 2024-09-04T19:36:45.829 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:45.827+0000 7f7d456c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:45.832 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:45.831+0000 7f7d456c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:45.833 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:45.831+0000 7f7d456c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:45.834 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:45.835+0000 7f7d456c47c0 -1 bdev(0x56037e376a80 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:36:45.834 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:45.835+0000 7f7d456c47c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-04T19:36:52.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-04T19:36:52.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:36:52.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:36:52.586 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:36:52.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:36:53.146 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:36:53.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:36:53.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:36:53.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:36:53.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:36:53.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:53.175+0000 7f9ec011a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:53.210 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:53.207+0000 7f9ec011a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:53.211 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:53.211+0000 7f9ec011a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:36:53.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:36:53.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:36:53.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:36:53.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:36:53.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:36:53.578 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:36:53.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:53.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:36:53.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:53.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:36:53.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:54.265 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:54.263+0000 7f9ec011a7c0 -1 Falling back to public interface 2024-09-04T19:36:54.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:54.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:54.972 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:36:54.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:36:54.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:54.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:36:55.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:56.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:56.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:56.374 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:36:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:36:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:36:56.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:57.243 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:36:57.243+0000 7f9ec011a7c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:36:57.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:57.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:57.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:36:57.799 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:36:57.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:57.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:36:58.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:36:59.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:36:59.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:36:59.420 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:36:59.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:36:59.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:36:59.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:36:59.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:00.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:00.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:00.855 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:37:00.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:37:00.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:00.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:01.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:02.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:02.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:02.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:37:02.427 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:37:02.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:02.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:02.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:03.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:04.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:05.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:05.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:05.408 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:37:05.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:37:05.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:05.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:05.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:06.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:06.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:06.839 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:37:06.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:37:06.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:06.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:07.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:08.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:08.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:08.279 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:37:08.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:37:08.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:08.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:08.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:09.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:09.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:09.733 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:37:09.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:37:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:10.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:11.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:11.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:11.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:37:11.222 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:37:11.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:11.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:11.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:12.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:12.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:12.908 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:37:12.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:37:12.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:12.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:13.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:37:14.986 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:37:14.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:14.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:15.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:16.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:16.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:16.836 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:37:16.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:37:16.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:16.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:17.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:18.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:18.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:18.691 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:37:18.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:37:18.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:18.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:19.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:20.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:20.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:20.391 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:37:20.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:37:20.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:20.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:21.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:22.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:22.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:22.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:37:22.402 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:37:22.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:22.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:37:22.833 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1413524638,v1:127.0.0.1:6811/1413524638] [v2:127.0.0.1:6812/1413524638,v1:127.0.0.1:6813/1413524638] exists,up ba86f7ae-7246-49a9-8ce8-4a54c46519f9 2024-09-04T19:37:22.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:37:22.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:37:22.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:37:22.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-04T19:37:22.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:37:22.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:37:22.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:37:22.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da7afb0b-0c66-4182-8136-2814d46ba32d 2024-09-04T19:37:22.840 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 da7afb0b-0c66-4182-8136-2814d46ba32d 2024-09-04T19:37:22.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 da7afb0b-0c66-4182-8136-2814d46ba32d' 2024-09-04T19:37:22.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:37:22.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDytthm9PEzMxAAb3Ez+33uqEOuMbQfPsXrPA== 2024-09-04T19:37:22.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDytthm9PEzMxAAb3Ez+33uqEOuMbQfPsXrPA=="}' 2024-09-04T19:37:22.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da7afb0b-0c66-4182-8136-2814d46ba32d -i td/ec-error-rollforward/2/new.json 2024-09-04T19:37:23.406 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:37:23.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-04T19:37:23.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDytthm9PEzMxAAb3Ez+33uqEOuMbQfPsXrPA== --osd-uuid da7afb0b-0c66-4182-8136-2814d46ba32d 2024-09-04T19:37:23.460 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:23.458+0000 7f6028cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:23.529 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:23.526+0000 7f6028cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:23.531 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:23.530+0000 7f6028cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:23.531 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:23.530+0000 7f6028cbd7c0 -1 bdev(0x55898672ea80 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:37:23.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:23.530+0000 7f6028cbd7c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-04T19:37:30.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-04T19:37:30.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:37:30.315 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:37:30.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:37:30.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:37:30.861 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:37:30.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:37:30.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:37:30.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:37:30.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:37:30.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:37:30.890 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:30.886+0000 7f9680e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:30.926 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:30.922+0000 7f9680e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:30.927 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:30.926+0000 7f9680e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:37:31.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:37:31.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:37:31.310 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:37:31.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:37:31.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:37:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:37:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:37:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:31.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:31.969 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:31.966+0000 7f9680e227c0 -1 Falling back to public interface 2024-09-04T19:37:32.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:32.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:32.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:37:32.744 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:37:32.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:32.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:33.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:34.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:34.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:34.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:37:34.185 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:37:34.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:34.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:34.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:34.950 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:37:34.946+0000 7f9680e227c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:37:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:37:35.608 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:37:35.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:35.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:36.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:37.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:37.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:37.268 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:37:37.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:37:37.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:37.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:37.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:38.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:38.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:38.672 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:37:38.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:37:38.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:39.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:40.307 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:37:40.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:37:40.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:40.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:40.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:41.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:41.891 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:37:41.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:41.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:37:41.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:41.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:42.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:43.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:43.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:43.364 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:37:43.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:37:43.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:43.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:43.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:44.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:44.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:44.902 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:37:44.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:37:44.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:44.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:45.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:46.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:46.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:46.447 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:37:46.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:37:46.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:46.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:46.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:47.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:47.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:47.985 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:37:47.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:37:47.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:47.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:48.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:49.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:49.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:49.435 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:37:49.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:37:49.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:49.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:49.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:50.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:50.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:50.888 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:37:50.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:37:50.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:50.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:51.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:52.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:52.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:52.600 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:37:52.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:37:52.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:52.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:53.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:37:54.533 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:37:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:55.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:56.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:56.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:56.225 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:37:56.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:37:56.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:56.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:56.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:37:57.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:37:57.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:37:57.919 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:37:57.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:37:57.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:37:57.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:37:59.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:00.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:00.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:00.077 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:38:00.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:38:00.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:00.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:38:00.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:01.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:01.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:01.644 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-04T19:38:01.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-04T19:38:01.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:01.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:38:02.062 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4082712510,v1:127.0.0.1:6819/4082712510] [v2:127.0.0.1:6820/4082712510,v1:127.0.0.1:6821/4082712510] exists,up da7afb0b-0c66-4182-8136-2814d46ba32d 2024-09-04T19:38:02.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:02.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:38:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:02.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:38:02.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 2024-09-04T19:38:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c26a2ea6-38e8-46c0-9093-1bb7f05b64c5' 2024-09-04T19:38:02.069 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 2024-09-04T19:38:02.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:38:02.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAat9hmJ5cuBRAA7VrjZvDOaWMN8ZEGSzKxrg== 2024-09-04T19:38:02.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAat9hmJ5cuBRAA7VrjZvDOaWMN8ZEGSzKxrg=="}' 2024-09-04T19:38:02.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 -i td/ec-error-rollforward/3/new.json 2024-09-04T19:38:02.640 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:38:02.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-04T19:38:02.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAat9hmJ5cuBRAA7VrjZvDOaWMN8ZEGSzKxrg== --osd-uuid c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 2024-09-04T19:38:02.689 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:02.686+0000 7f7f3c50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:02.710+0000 7f7f3c50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:02.713 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:02.710+0000 7f7f3c50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:02.713 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:02.710+0000 7f7f3c50a7c0 -1 bdev(0x55fcea6f2a80 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-04T19:38:02.713 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:02.710+0000 7f7f3c50a7c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-04T19:38:09.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-04T19:38:09.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:38:09.647 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-04T19:38:09.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-04T19:38:09.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:38:10.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-04T19:38:10.185 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:38:10.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:38:10.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:38:10.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:38:10.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:38:10.215 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:10.214+0000 7f57fb0027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:10.217 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:10.214+0000 7f57fb0027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:10.218 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:10.214+0000 7f57fb0027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:38:10.609 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:38:10.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:10.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:38:10.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:10.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:11.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:11.281 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:11.278+0000 7f57fb0027c0 -1 Falling back to public interface 2024-09-04T19:38:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:38:12.109 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:38:12.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:12.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:12.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:13.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:13.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:13.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:38:13.530 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:38:13.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:13.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:13.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:14.024 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:14.022+0000 7f57fb0027c0 -1 osd.3 0 log_to_monitors true 2024-09-04T19:38:14.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:14.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:14.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:38:14.955 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:38:14.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:14.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:15.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:16.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:16.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:16.467 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:38:16.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:38:16.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:16.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:17.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:18.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:18.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:18.019 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:38:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:38:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:18.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:18.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:19.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:19.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:19.428 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:38:19.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:38:19.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:19.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:20.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:21.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:21.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:21.142 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:38:21.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:38:21.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:21.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:21.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:22.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:22.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:22.649 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:38:22.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:38:22.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:22.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:23.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:24.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:24.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:24.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:38:24.108 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:38:24.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:24.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:24.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:25.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:25.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:25.567 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:38:25.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:38:25.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:25.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:26.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:27.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:27.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:27.082 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:38:27.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:38:27.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:27.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:27.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:28.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:28.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:28.668 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:38:28.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:38:28.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:28.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:29.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:30.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:30.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:30.357 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:38:30.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:38:30.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:30.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:31.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:32.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:32.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:32.602 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:38:32.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:38:32.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:32.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:33.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:34.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:34.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:34.460 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:38:34.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:38:34.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:34.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:35.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:36.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:36.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:36.361 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:38:36.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:38:36.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:36.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:37.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:38:38.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:38:38.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:38.551 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:38:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:38:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:39.099 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3666735812,v1:127.0.0.1:6827/3666735812] [v2:127.0.0.1:6828/3666735812,v1:127.0.0.1:6829/3666735812] exists,up c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 2024-09-04T19:38:39.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:39.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:39.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:39.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-04T19:38:39.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-04T19:38:40.504 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'ec' already exists 2024-09-04T19:38:40.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-04T19:38:42.091 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-04T19:38:42.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 89267 2024-09-04T19:38:42.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=92047 2024-09-04T19:38:42.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-04T19:38:42.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-04T19:38:43.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 92068' 2024-09-04T19:38:43.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-04T19:38:43.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 92069' 2024-09-04T19:38:43.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-04T19:38:43.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 92070' 2024-09-04T19:38:43.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-04T19:38:43.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-04T19:38:44.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-04T19:38:44.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:38:44.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:38:44.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:38:44.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:38:44.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:38:44.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:38:44.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 92047 92068 92069 92070 2024-09-04T19:38:44.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-04T19:38:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-04T19:38:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-04T19:38:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:38:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:38:44.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:38:44.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:38:44.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:38:44.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:38:44.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:38:44.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:38:44.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-04T19:38:44.419 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:38:44.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:38:44.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:38:44.444 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:44.441+0000 7f075ee027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:44.447 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:44.445+0000 7f075ee027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:44.448 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:44.445+0000 7f075ee027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:38:44.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:44.842 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:38:44.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:38:44.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:44.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:38:45.261 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/50983507,v1:127.0.0.1:6803/50983507] [v2:127.0.0.1:6804/50983507,v1:127.0.0.1:6805/50983507] exists,up 2f6a11dd-e321-4df6-969d-fb779dde24e5 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:38:45.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:38:45.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:38:45.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:38:45.267 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:38:45.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:38:45.268 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-04T19:38:45.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:38:45.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:38:45.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:38:45.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:38:45.298 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:45.293+0000 7f54a80d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:45.303 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:45.301+0000 7f54a80d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:45.304 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:45.301+0000 7f54a80d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:38:45.673 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:38:45.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:45.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:38:45.993 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:45.989+0000 7f075ee027c0 -1 Falling back to public interface 2024-09-04T19:38:46.157 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1413524638,v1:127.0.0.1:6811/1413524638] [v2:127.0.0.1:6812/1413524638,v1:127.0.0.1:6813/1413524638] exists,up ba86f7ae-7246-49a9-8ce8-4a54c46519f9 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:38:46.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:46.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:46.161 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:46.161 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:46.161 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:46.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:46.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:46.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:38:46.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:38:46.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:38:46.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:38:46.163 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:38:46.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:38:46.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-04T19:38:46.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:38:46.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:38:46.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:38:46.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:38:46.191 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:46.189+0000 7fa4e171a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:46.193 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:46.189+0000 7fa4e171a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:46.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:46.193+0000 7fa4e171a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:46.352 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:46.349+0000 7f54a80d87c0 -1 Falling back to public interface 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:38:46.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:46.601 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:38:46.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:38:46.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:46.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:38:47.130 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4082712510,v1:127.0.0.1:6819/4082712510] [v2:127.0.0.1:6820/4082712510,v1:127.0.0.1:6821/4082712510] exists,up da7afb0b-0c66-4182-8136-2814d46ba32d 2024-09-04T19:38:47.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:38:47.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' 2024-09-04T19:38:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-04T19:38:47.136 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:38:47.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-04T19:38:47.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=09fcdfc9-6106-440e-a002-4791c3a92734 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-04T19:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-04T19:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:38:47.163 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:47.161+0000 7fa91c9e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:47.191 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:47.189+0000 7fa91c9e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:47.192 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:47.189+0000 7fa91c9e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:38:47.493 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:47.489+0000 7fa4e171a7c0 -1 Falling back to public interface 2024-09-04T19:38:47.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-04T19:38:47.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:38:47.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:38:47.600 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:38:47.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:38:47.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:38:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:38:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:38:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:38:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:38:48.015 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3666735812,v1:127.0.0.1:6827/3666735812] [v2:127.0.0.1:6828/3666735812,v1:127.0.0.1:6829/3666735812] exists,up c26a2ea6-38e8-46c0-9093-1bb7f05b64c5 2024-09-04T19:38:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:38:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:38:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:38:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-04T19:38:48.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:38:48.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:38:48.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:38:48.017 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:38:48.017 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:38:48.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:38:48.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:38:48.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:38:48.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:38:48.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:38:48.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:38:48.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:38:48.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:38:48.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:38:48.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:38:48.563 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:38:49.053 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:49.049+0000 7fa91c9e47c0 -1 Falling back to public interface 2024-09-04T19:38:50.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:50.149+0000 7f075ee027c0 -1 osd.0 25 log_to_monitors true 2024-09-04T19:38:50.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:50.725+0000 7f54a80d87c0 -1 osd.1 25 log_to_monitors true 2024-09-04T19:38:52.005 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:52.001+0000 7fa4e171a7c0 -1 osd.2 24 log_to_monitors true 2024-09-04T19:38:53.210 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:38:53.205+0000 7fa91c9e47c0 -1 osd.3 25 log_to_monitors true 2024-09-04T19:38:53.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084291 2024-09-04T19:38:53.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084291 2024-09-04T19:38:53.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291' 2024-09-04T19:38:53.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:38:53.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:38:53.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018883 2024-09-04T19:38:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018883 2024-09-04T19:38:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291 1-128849018883' 2024-09-04T19:38:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:38:53.995 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:38:54.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-09-04T19:38:54.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-09-04T19:38:54.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291 1-128849018883 2-146028888067' 2024-09-04T19:38:54.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:38:54.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-04T19:38:54.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=3 2024-09-04T19:38:54.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 3 2024-09-04T19:38:54.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084291 1-128849018883 2-146028888067 3-3' 2024-09-04T19:38:54.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:38:54.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084291 2024-09-04T19:38:54.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:38:54.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:38:54.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084291 2024-09-04T19:38:54.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:38:54.481 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 120259084291 2024-09-04T19:38:54.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084291 2024-09-04T19:38:54.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2024-09-04T19:38:54.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:38:55.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084291 -lt 120259084291 2024-09-04T19:38:55.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:38:55.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018883 2024-09-04T19:38:55.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:38:55.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:38:55.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018883 2024-09-04T19:38:55.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:38:55.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018883 2024-09-04T19:38:55.025 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 128849018883 2024-09-04T19:38:55.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018883' 2024-09-04T19:38:55.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:38:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018883 -lt 128849018883 2024-09-04T19:38:55.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:38:55.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888067 2024-09-04T19:38:55.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:38:55.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:38:55.486 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:38:55.486 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888067 2024-09-04T19:38:55.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-09-04T19:38:55.487 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 146028888067 2024-09-04T19:38:55.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-09-04T19:38:55.488 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:38:55.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888067 -lt 146028888067 2024-09-04T19:38:55.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:38:55.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-3 2024-09-04T19:38:55.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:38:55.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-04T19:38:55.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-3 2024-09-04T19:38:55.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:38:55.931 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 3 2024-09-04T19:38:55.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=3 2024-09-04T19:38:55.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 3' 2024-09-04T19:38:55.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-04T19:38:56.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 3 2024-09-04T19:38:56.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:38:57.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:38:57.354 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-04T19:38:57.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 3 -lt 3 2024-09-04T19:38:57.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:38:57.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:38:57.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:38:58.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-04T19:38:58.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:38:58.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:38:58.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-04T19:38:58.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:38:58.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:38:58.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:38:59.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-04T19:38:59.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:38:59.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:38:59.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:38:59.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:38:59.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:38:59.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:38:59.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:38:59.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:38:59.444 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:38:59.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:38:59.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:38:59.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:38:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:38:59.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:38:59.447 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:38:59.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:38:59.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-04T19:38:59.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:38:59.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:59.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:59.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84609 2024-09-04T19:38:59.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:38:59.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:38:59.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-04T19:38:59.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ec-error-rollforward 0 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:38:59.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:38:59.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:38:59.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:38:59.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:38:59.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:38:59.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:38:59.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:38:59.473 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:38:59.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:38:59.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:38:59.475 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:38:59.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-04T19:38:59.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-04T19:38:59.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:38:59.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:59.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84609 2024-09-04T19:38:59.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84609 2024-09-04T19:38:59.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:38:59.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:38:59.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-04T19:38:59.479 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-04T19:38:59.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-04T19:38:59.531 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-04T19:38:59.531 DEBUG:teuthology.orchestra.run.smithi174:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-04T19:38:59.582 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-04T19:38:59.587 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-04T19:38:59.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:38:59.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:38:59.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-bench 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-04T19:38:59.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-04T19:38:59.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=e1dec81e-2ab6-4b0e-82c1-6ec5335427b8 --auth-supported=none ' 2024-09-04T19:38:59.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-04T19:38:59.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-04T19:38:59.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-04T19:38:59.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:38:59.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:38:59.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:38:59.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:38:59.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:38:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:38:59.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:38:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:38:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:38:59.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-04T19:38:59.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:38:59.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:59.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:38:59.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94607 2024-09-04T19:38:59.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:38:59.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:38:59.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-09-04T19:38:59.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:38:59.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:59.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:38:59.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94607 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-04T19:38:59.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-09-04T19:38:59.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:38:59.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:38:59.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:38:59.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-09-04T19:38:59.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:38:59.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94607/$cluster-$name.asok' 2024-09-04T19:38:59.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.94607/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:39:00.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:39:00.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:39:00.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:39:00.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:39:00.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:39:00.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:39:00.245 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:39:00.245 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:39:00.245 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:39:00.245 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:39:00.246 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:39:00.246 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:39:00.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94607/ceph-mon.a.asok 2024-09-04T19:39:00.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:39:00.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94607/ceph-mon.a.asok config get fsid 2024-09-04T19:39:00.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:39:00.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:39:00.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:39:00.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:39:00.362 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94607/ceph-mon.a.asok 2024-09-04T19:39:00.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:39:00.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94607/ceph-mon.a.asok config get mon_host 2024-09-04T19:39:00.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-09-04T19:39:00.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:39:01.381 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94607/$cluster-$name.asok' 2024-09-04T19:39:01.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:39:01.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.94607/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:39:01.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-04T19:39:01.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e1dec81e-2ab6-4b0e-82c1-6ec5335427b8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:39:01.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94607/$cluster-$name.asok' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94607/$cluster-$name.asok' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:39:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:39:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-09-04T19:39:01.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:39:01.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e35eb8a6-b422-414e-ba27-dd8bb4587981 2024-09-04T19:39:01.422 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 e35eb8a6-b422-414e-ba27-dd8bb4587981 2024-09-04T19:39:01.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e35eb8a6-b422-414e-ba27-dd8bb4587981' 2024-09-04T19:39:01.423 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:39:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVt9hmj2MJGhAA5yLrY7MBckzsr2u3Ag9Qnw== 2024-09-04T19:39:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVt9hmj2MJGhAA5yLrY7MBckzsr2u3Ag9Qnw=="}' 2024-09-04T19:39:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e35eb8a6-b422-414e-ba27-dd8bb4587981 -i td/osd-bench/0/new.json 2024-09-04T19:39:01.729 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:39:01.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-09-04T19:39:01.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e1dec81e-2ab6-4b0e-82c1-6ec5335427b8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.94607/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVt9hmj2MJGhAA5yLrY7MBckzsr2u3Ag9Qnw== --osd-uuid e35eb8a6-b422-414e-ba27-dd8bb4587981 2024-09-04T19:39:01.774 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:01.769+0000 7f16b594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:01.789 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:01.785+0000 7f16b594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:01.791 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:01.789+0000 7f16b594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:01.791 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:01.789+0000 7f16b594a7c0 -1 bdev(0x5636a47d0a80 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:39:01.791 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:01.789+0000 7f16b594a7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-04T19:39:08.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-04T19:39:08.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:39:08.425 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:39:08.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:39:08.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:39:08.964 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:39:08.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:39:08.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e1dec81e-2ab6-4b0e-82c1-6ec5335427b8 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.94607/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:39:08.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:39:08.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:39:08.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:39:08.993 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:08.989+0000 7f81832757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:08.996 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:08.993+0000 7f81832757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:08.997 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:08.993+0000 7f81832757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:39:09.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:39:09.399 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:39:09.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:09.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:09.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:10.026 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:10.021+0000 7f81832757c0 -1 Falling back to public interface 2024-09-04T19:39:10.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:10.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:10.834 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:39:10.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:39:10.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:10.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:11.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:12.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:12.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:12.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:39:12.243 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:39:12.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:12.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:12.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:12.863 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:39:12.861+0000 7f81832757c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:39:13.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:13.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:13.662 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:39:13.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:39:13.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:13.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:14.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:15.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:15.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:15.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:39:15.102 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:39:15.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:15.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:15.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:16.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:16.547 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:39:16.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:16.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:39:16.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:16.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:17.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:18.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:18.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:18.087 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:39:18.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:39:18.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:18.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:18.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:19.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:19.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:19.526 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:39:19.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:39:19.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:19.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:20.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:21.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:21.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:21.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:39:21.086 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:39:21.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:21.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:21.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:22.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:22.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:22.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:39:22.556 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:39:22.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:22.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:23.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:24.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:24.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:24.126 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:39:24.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:39:24.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:24.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:24.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:25.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:25.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:25.549 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:39:25.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:39:25.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:25.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:26.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:27.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:27.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:27.166 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:39:27.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:39:27.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:27.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:27.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:28.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:28.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:28.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:39:28.677 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:39:28.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:28.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:29.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:30.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:30.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:30.828 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:39:30.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:39:30.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:30.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:32.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:32.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:32.555 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:39:32.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:39:32.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:32.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:33.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:34.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:34.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:34.658 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:39:34.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:39:34.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:34.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:35.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:36.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:36.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:36.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:39:36.344 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:39:36.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:36.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:37.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:39:38.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:39:38.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:39:38.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:39:38.497 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:39:38.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:39:38.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:39:38.921 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3790406936,v1:127.0.0.1:6803/3790406936] [v2:127.0.0.1:6804/3790406936,v1:127.0.0.1:6805/3790406936] exists,up e35eb8a6-b422-414e-ba27-dd8bb4587981 2024-09-04T19:39:38.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:39:38.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:39:38.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:39:38.922 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-04T19:39:38.922 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-04T19:39:38.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-04T19:39:38.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-04T19:39:38.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-04T19:39:38.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-04T19:39:38.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-04T19:39:38.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-04T19:39:38.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-04T19:39:38.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-04T19:39:38.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=osd_bench_max_block_size 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-04T19:39:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-04T19:39:39.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-04T19:39:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-04T19:39:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-09-04T19:39:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-04T19:39:39.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=bench_small_size_max_iops 2024-09-04T19:39:39.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-04T19:39:39.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-04T19:39:39.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-04T19:39:39.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-04T19:39:39.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-04T19:39:39.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-04T19:39:39.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-04T19:39:39.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-04T19:39:39.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/osd-bench 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-04T19:39:39.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-04T19:39:39.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-04T19:39:39.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-04T19:39:39.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-04T19:39:39.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-04T19:39:39.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-04T19:40:33.977 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:40:33.977 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_written": 1073741824, 2024-09-04T19:40:33.977 INFO:tasks.workunit.client.0.smithi174.stdout: "blocksize": 4194304, 2024-09-04T19:40:33.978 INFO:tasks.workunit.client.0.smithi174.stdout: "elapsed_sec": 26.482159171999999, 2024-09-04T19:40:33.978 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_per_sec": 40545856.439654812, 2024-09-04T19:40:33.978 INFO:tasks.workunit.client.0.smithi174.stdout: "iops": 9.6668854807984381 2024-09-04T19:40:33.978 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:40:34.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-04T19:40:34.423 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_written": 10, 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout: "blocksize": 14456, 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout: "elapsed_sec": 0.033447473999999998, 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_per_sec": 298.9762395809023, 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout: "iops": 0.020681809600228439 2024-09-04T19:40:34.424 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:40:34.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_written": 1, 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout: "blocksize": 14456, 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout: "elapsed_sec": 0.032013316999999999, 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout: "bytes_per_sec": 31.237000526999438, 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout: "iops": 0.0021608329086192195 2024-09-04T19:40:34.758 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-04T19:40:34.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-04T19:40:34.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-04T19:40:34.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:40:34.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-04T19:40:34.782 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:40:34.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:40:34.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:40:34.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:40:34.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:40:35.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:40:35.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:40:35.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:40:35.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:40:35.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:40:35.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:40:35.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:40:35.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:40:35.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:40:35.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:40:35.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:40:35.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-04T19:40:35.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:40:35.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:40:35.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94607 2024-09-04T19:40:35.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-bench 0 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:40:35.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:40:35.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:40:35.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:40:35.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:40:35.139 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:40:35.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:40:35.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:40:35.141 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:40:35.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:40:35.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:40:35.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:40:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-04T19:40:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-04T19:40:35.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:40:35.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94607 2024-09-04T19:40:35.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94607 2024-09-04T19:40:35.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:40:35.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:40:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-04T19:40:35.148 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-04T19:40:35.148 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-04T19:40:35.208 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-04T19:40:35.209 DEBUG:teuthology.orchestra.run.smithi174:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-04T19:40:35.263 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-04T19:40:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-bluefs-volume-ops 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-04T19:40:35.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-04T19:40:35.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-04T19:40:35.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-04T19:40:35.273 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_bluestore2 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_bluestore_expand' 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:40:35.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:40:35.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:40:35.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:40:35.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:40:35.277 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:40:35.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:40:35.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:40:35.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:40:35.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:40:35.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:40:35.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:40:35.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:40:35.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:40:35.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:40:35.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:40:35.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:35.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:40:35.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:40:35.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:40:35.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-04T19:40:35.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:40:35.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:35.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96820 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-04T19:40:35.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-04T19:40:35.290 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-04T19:40:35.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none ' 2024-09-04T19:40:35.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-04T19:40:35.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-04T19:40:35.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-04T19:40:35.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-04T19:40:35.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:35.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:40:35.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:40:35.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:40:35.703 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:40:35.703 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.703 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:35.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:40:35.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:40:35.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:40:35.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:40:35.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:40:35.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:40:35.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:40:35.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:40:35.899 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:40:35.899 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:40:35.899 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:40:35.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get fsid 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:40:36.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:40:36.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get mon_host 2024-09-04T19:40:36.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-04T19:40:36.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:36.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:40:36.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:40:36.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:40:36.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-04T19:40:36.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:40:37.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:40:37.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:40:37.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:40:37.000 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:40:37.000 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:37.001 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:37.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:40:37.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:40:37.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:40:37.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:40:37.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:40:37.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:40:37.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:40:37.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:40:37.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:40:37.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:40:37.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 256dd40b-59b2-408c-b6b4-c2218bb84138' 2024-09-04T19:40:37.037 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:40:37.037 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:40:37.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1t9hmwoMEAxAAHw9oJjEnOalDtbXS8UZN/w== 2024-09-04T19:40:37.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1t9hmwoMEAxAAHw9oJjEnOalDtbXS8UZN/w=="}' 2024-09-04T19:40:37.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 256dd40b-59b2-408c-b6b4-c2218bb84138 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:40:37.333 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:40:37.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:40:37.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1t9hmwoMEAxAAHw9oJjEnOalDtbXS8UZN/w== --osd-uuid 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:40:37.381 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:37.375+0000 7f049625b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:37.401 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:37.395+0000 7f049625b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:37.403 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:37.399+0000 7f049625b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:37.403 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:37.399+0000 7f049625b7c0 -1 bdev(0x5631bb78ea80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:40:37.403 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:37.399+0000 7f049625b7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-04T19:40:47.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-04T19:40:47.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:40:47.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:40:47.002 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:40:47.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:40:47.554 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:40:47.586 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:47.579+0000 7f23651ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:47.613 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:47.607+0000 7f23651ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:47.615 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:47.611+0000 7f23651ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:40:47.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:47.970 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:40:47.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:40:47.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:47.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:48.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:49.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:49.127+0000 7f23651ce7c0 -1 Falling back to public interface 2024-09-04T19:40:49.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:49.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:49.427 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:40:49.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:40:49.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:49.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:50.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:50.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:50.896 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:40:50.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:40:50.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:50.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:51.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:52.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:52.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:52.296 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:40:52.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:40:52.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:52.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:52.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:53.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:53.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:53.722 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:40:53.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:40:53.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:53.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:54.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:54.481 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:40:54.475+0000 7f23651ce7c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:40:55.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:55.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:55.120 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:40:55.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:40:55.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:55.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:55.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:56.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:56.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:56.622 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:40:56.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:40:56.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:56.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:57.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:58.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:58.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:58.052 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:40:58.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:40:58.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:58.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:58.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:40:59.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:40:59.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:40:59.488 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:40:59.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:40:59.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:40:59.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:40:59.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:00.910 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:41:00.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:41:00.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:00.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:01.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:02.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:02.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:02.451 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:41:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:41:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:02.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:02.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:03.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:03.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:03.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:41:03.892 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:41:03.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:03.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:04.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:05.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:05.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:05.398 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:41:05.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:41:05.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:05.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:05.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:06.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:06.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:06.907 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:41:06.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:41:06.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:06.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:07.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:08.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:08.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:08.517 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:41:08.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:41:08.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:08.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:09.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:10.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:10.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:10.062 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:41:10.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:41:10.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:10.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:10.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:11.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:11.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:11.893 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:41:11.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:41:11.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:11.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:12.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:13.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:13.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:13.419 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:41:13.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:41:13.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:13.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:14.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:15.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:15.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:15.327 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:41:15.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:41:15.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:15.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:15.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:16.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:16.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:16.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-04T19:41:16.910 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-04T19:41:16.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:16.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:41:17.397 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3653085089,v1:127.0.0.1:6803/3653085089] [v2:127.0.0.1:6804/3653085089,v1:127.0.0.1:6805/3653085089] exists,up 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:41:17.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:41:17.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:41:17.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:41:17.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:41:17.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=98286 2024-09-04T19:41:17.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-04T19:41:17.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:41:17.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:41:17.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-04T19:41:17.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:41:17.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:41:17.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 30112775-75b5-463d-83a5-bf8aa20d94aa' 2024-09-04T19:41:17.405 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:41:17.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:41:17.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdt9hmu5wwGRAAo8/N5HszoC7DKjU5KR562g== 2024-09-04T19:41:17.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdt9hmu5wwGRAAo8/N5HszoC7DKjU5KR562g=="}' 2024-09-04T19:41:17.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30112775-75b5-463d-83a5-bf8aa20d94aa -i td/osd-bluefs-volume-ops/1/new.json 2024-09-04T19:41:17.905 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:41:17.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-04T19:41:17.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdt9hmu5wwGRAAo8/N5HszoC7DKjU5KR562g== --osd-uuid 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:41:17.950 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:17.947+0000 7fede8e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:17.955 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:17.955+0000 7fede8e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:17.956 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:17.955+0000 7fede8e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:17.956 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:17.955+0000 7fede8e647c0 -1 bdev(0x558a3ea32a80 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-04T19:41:17.956 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:17.955+0000 7fede8e647c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-04T19:41:27.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-04T19:41:27.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:41:27.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-04T19:41:27.439 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-04T19:41:27.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:41:27.999 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:41:28.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-04T19:41:28.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:41:28.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:41:28.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:41:28.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:41:28.031 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:28.031+0000 7f3159e787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:28.041 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:28.039+0000 7f3159e787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:28.042 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:28.039+0000 7f3159e787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:28.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-04T19:41:28.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:41:28.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:41:28.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:41:28.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:41:28.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:28.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:41:28.399 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:41:28.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:28.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:28.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:29.797 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:29.795+0000 7f3159e787c0 -1 Falling back to public interface 2024-09-04T19:41:29.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:29.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:29.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:41:29.864 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:41:29.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:29.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:30.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:31.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:31.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:31.301 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:41:31.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:41:31.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:31.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:31.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:32.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:32.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:32.784 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:41:32.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:41:32.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:32.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:33.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:34.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:34.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:34.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:41:34.199 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:41:34.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:34.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:34.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:34.897 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:34.895+0000 7f3159e787c0 -1 osd.1 0 log_to_monitors true 2024-09-04T19:41:35.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:35.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:35.639 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:41:35.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:41:35.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:35.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:36.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:37.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:37.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:37.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:41:37.391 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:41:37.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:37.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:37.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:38.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:38.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:38.983 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:41:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:41:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:38.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:39.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:40.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:40.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:40.386 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:41:40.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:41:40.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:40.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:40.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:41.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:41.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:41.924 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:41:41.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:41:41.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:41.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:42.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:43.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:43.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:43.398 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:41:43.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:41:43.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:43.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:43.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:44.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:44.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:44.960 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:41:44.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:41:44.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:45.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:46.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:46.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:46.426 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:41:46.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:41:46.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:46.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:47.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:48.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:48.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:48.032 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:41:48.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:41:48.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:48.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:48.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:49.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:49.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:49.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:41:49.737 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:41:49.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:49.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:50.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:51.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:51.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:51.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:41:51.806 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:41:51.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:51.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:53.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:54.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:54.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:54.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:41:54.035 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:41:54.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:54.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:54.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:55.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:55.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:55.815 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:41:55.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:41:55.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:55.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:56.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:41:57.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:41:57.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:41:57.826 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:41:57.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:41:57.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:41:57.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:41:58.249 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/536341311,v1:127.0.0.1:6811/536341311] [v2:127.0.0.1:6812/536341311,v1:127.0.0.1:6813/536341311] exists,up 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:41:58.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:41:58.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:41:58.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:41:58.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-04T19:41:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=100840 2024-09-04T19:41:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-04T19:41:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:41:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:41:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:41:58.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:41:58.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:41:58.255 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:41:58.255 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:41:58.255 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:41:58.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:41:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:41:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:41:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:41:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-04T19:41:58.257 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:41:58.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:41:58.258 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:41:58.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b' 2024-09-04T19:41:58.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:41:58.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGuNhmZCt3EBAAuWt6t50HPK1+32N3Edg0hQ== 2024-09-04T19:41:58.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGuNhmZCt3EBAAuWt6t50HPK1+32N3Edg0hQ=="}' 2024-09-04T19:41:58.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b -i td/osd-bluefs-volume-ops/2/new.json 2024-09-04T19:41:58.764 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:41:58.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-04T19:41:58.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGuNhmZCt3EBAAuWt6t50HPK1+32N3Edg0hQ== --osd-uuid 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:41:58.814 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:58.814+0000 7fe63d7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:58.817 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:58.814+0000 7fe63d7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:58.819 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:58.818+0000 7fe63d7b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:41:58.819 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:58.818+0000 7fe63d7b37c0 -1 bdev(0x55715f700a80 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-04T19:41:58.819 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:41:58.818+0000 7fe63d7b37c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-04T19:42:08.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-04T19:42:08.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:42:08.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-04T19:42:08.547 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-04T19:42:08.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:42:09.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-04T19:42:09.093 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:42:09.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:42:09.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:42:09.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:42:09.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:42:09.124 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:09.122+0000 7f3eb16777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:09.125 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:09.122+0000 7f3eb16777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:09.127 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:09.126+0000 7f3eb16777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:09.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-04T19:42:09.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:42:09.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:42:09.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:09.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:10.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:10.630 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:10.626+0000 7f3eb16777c0 -1 Falling back to public interface 2024-09-04T19:42:11.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:11.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:11.013 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:42:11.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:42:11.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:11.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:11.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:12.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:12.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:12.417 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:42:12.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:42:12.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:12.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:12.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:13.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:13.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:13.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:42:13.825 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:42:13.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:13.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:14.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:15.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:15.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:15.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:42:15.239 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:42:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:15.621 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:15.618+0000 7f3eb16777c0 -1 osd.2 0 log_to_monitors true 2024-09-04T19:42:15.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:16.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:16.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:16.647 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:42:16.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:42:16.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:16.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:17.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:18.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:18.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:18.112 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:42:18.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:42:18.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:18.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:18.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:19.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:19.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:19.671 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:42:19.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:42:19.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:19.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:20.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:21.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:21.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:21.080 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:42:21.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:42:21.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:21.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:21.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:22.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:22.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:22.660 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:42:22.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:42:22.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:22.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:23.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:24.071 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:42:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:42:24.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:24.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:24.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:25.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:25.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:25.531 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:42:25.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:42:25.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:25.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:26.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:27.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:27.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:27.020 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:42:27.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:42:27.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:27.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:27.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:28.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:28.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:28.791 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:42:28.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:42:28.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:28.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:29.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:30.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:30.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:30.552 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:42:30.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:42:30.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:30.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:31.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:32.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:32.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:32.326 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:42:32.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:42:32.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:32.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:33.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:34.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:34.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:34.010 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:42:34.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:42:34.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:34.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:34.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:35.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:35.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:35.662 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:42:35.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:42:35.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:35.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:36.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:37.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:37.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:37.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:42:37.099 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:42:37.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:37.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:37.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:38.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:38.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:38.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-04T19:42:38.906 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-04T19:42:38.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:38.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:42:39.316 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/558483087,v1:127.0.0.1:6819/558483087] [v2:127.0.0.1:6820/558483087,v1:127.0.0.1:6821/558483087] exists,up 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:42:39.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:42:39.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:42:39.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:42:39.317 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-04T19:42:39.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=103369 2024-09-04T19:42:39.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-04T19:42:39.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:42:39.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:42:39.322 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:42:39.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 59d144bb-00ee-453e-9caf-6b562763e52c' 2024-09-04T19:42:39.325 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:42:39.326 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:42:39.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvuNhm4iR8FBAAspsD6umRRyuj10f0u+H1og== 2024-09-04T19:42:39.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvuNhm4iR8FBAAspsD6umRRyuj10f0u+H1og=="}' 2024-09-04T19:42:39.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59d144bb-00ee-453e-9caf-6b562763e52c -i td/osd-bluefs-volume-ops/3/new.json 2024-09-04T19:42:39.911 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:42:39.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-04T19:42:39.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvuNhm4iR8FBAAspsD6umRRyuj10f0u+H1og== --osd-uuid 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:42:39.960 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:39.958+0000 7f47206147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:39.962 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:39.962+0000 7f47206147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:39.964 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:39.962+0000 7f47206147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:39.964 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:39.962+0000 7f47206147c0 -1 bdev(0x56075616ca80 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-04T19:42:39.964 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:39.962+0000 7f47206147c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-04T19:42:49.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-04T19:42:49.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:42:49.450 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-04T19:42:49.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-04T19:42:49.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:42:50.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-04T19:42:50.008 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:42:50.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:42:50.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:42:50.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:42:50.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:42:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:50.034+0000 7f9720f107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:50.055 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:50.054+0000 7f9720f107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:50.057 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:50.054+0000 7f9720f107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:42:50.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-04T19:42:50.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:42:50.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:42:50.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:42:50.444 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:42:50.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:42:50.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:50.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:42:50.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:50.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:50.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:51.821 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:51.818+0000 7f9720f107c0 -1 Falling back to public interface 2024-09-04T19:42:51.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:51.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:51.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:42:51.888 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:42:51.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:51.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:52.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:53.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:53.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:53.317 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:42:53.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:42:53.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:53.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:53.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:54.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:54.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:54.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:42:54.801 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:42:54.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:54.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:55.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:56.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:56.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:56.242 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:42:56.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:42:56.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:56.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:56.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:57.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:42:57.189+0000 7f9720f107c0 -1 osd.3 0 log_to_monitors true 2024-09-04T19:42:57.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:57.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:57.662 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:42:57.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:42:57.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:57.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:58.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:42:59.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:42:59.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:42:59.365 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:42:59.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:42:59.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:42:59.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:42:59.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:00.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:00.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:00.810 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:43:00.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:43:00.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:00.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:01.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:02.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:02.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:02.321 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:43:02.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:43:02.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:02.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:02.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:03.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:03.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:03.852 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:43:03.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:43:03.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:03.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:04.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:05.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:05.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:05.393 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-04T19:43:05.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-04T19:43:05.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:05.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:05.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:06.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:06.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:06.818 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-04T19:43:06.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-04T19:43:06.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:06.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:07.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:08.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:08.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:08.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-04T19:43:08.319 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-04T19:43:08.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:08.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:08.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:09.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:09.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:09.839 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-04T19:43:09.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-04T19:43:09.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:09.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:10.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:11.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:11.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:11.941 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-04T19:43:11.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-04T19:43:11.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:11.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:12.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:13.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-04T19:43:13.529 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-04T19:43:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:15.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:15.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:15.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-04T19:43:15.221 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-04T19:43:15.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:15.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:15.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:16.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:16.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:16.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-04T19:43:16.938 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-04T19:43:16.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:16.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:17.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:18.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:18.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:18.616 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-04T19:43:18.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-04T19:43:18.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:18.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:19.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:43:20.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:43:20.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:43:20.315 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-04T19:43:20.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-04T19:43:20.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:43:20.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:43:20.746 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/17220531,v1:127.0.0.1:6827/17220531] [v2:127.0.0.1:6828/17220531,v1:127.0.0.1:6829/17220531] exists,up 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:43:20.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:43:20.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:43:20.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:43:20.747 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-04T19:43:20.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=105923 2024-09-04T19:43:20.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-04T19:43:25.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-04T19:43:25.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-04T19:43:26.256 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'foo' already exists 2024-09-04T19:43:26.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:43:27.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-04T19:43:27.421 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:43:27.421 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-04T19:43:27.422 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_107288 2024-09-04T19:43:27.422 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:43:27.422 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:43:28.422 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 50 34 0.132798 0.132812 0.311338 0.424028 2024-09-04T19:43:29.422 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 93 77 0.150374 0.167969 0.291853 0.369233 2024-09-04T19:43:30.422 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 154 138 0.179667 0.238281 0.307233 0.33597 2024-09-04T19:43:31.422 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 217 201 0.196265 0.246094 0.276645 0.312337 2024-09-04T19:43:32.422 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 271 255 0.199196 0.210938 0.252679 0.300748 2024-09-04T19:43:33.422 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 335 319 0.207659 0.25 0.233906 0.293469 2024-09-04T19:43:34.422 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 395 379 0.211469 0.234375 0.259936 0.289792 2024-09-04T19:43:35.422 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 459 443 0.216281 0.25 0.310523 0.285293 2024-09-04T19:43:36.423 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 511 495 0.214816 0.203125 0.280984 0.282673 2024-09-04T19:43:37.423 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 576 560 0.218721 0.253906 0.241999 0.282865 2024-09-04T19:43:38.423 INFO:tasks.workunit.client.0.smithi174.stdout: 11 16 641 625 0.221917 0.253906 0.21823 0.279336 2024-09-04T19:43:39.423 INFO:tasks.workunit.client.0.smithi174.stdout: 12 16 705 689 0.224255 0.25 0.235069 0.274927 2024-09-04T19:43:40.423 INFO:tasks.workunit.client.0.smithi174.stdout: 13 16 766 750 0.225332 0.238281 0.300695 0.27435 2024-09-04T19:43:41.423 INFO:tasks.workunit.client.0.smithi174.stdout: 14 16 833 817 0.227929 0.261719 0.230632 0.272343 2024-09-04T19:43:42.423 INFO:tasks.workunit.client.0.smithi174.stdout: 15 16 898 882 0.229659 0.253906 0.261716 0.270613 2024-09-04T19:43:43.423 INFO:tasks.workunit.client.0.smithi174.stdout: 16 16 963 947 0.231173 0.253906 0.259131 0.26916 2024-09-04T19:43:44.424 INFO:tasks.workunit.client.0.smithi174.stdout: 17 16 1011 995 0.228603 0.1875 0.268036 0.271561 2024-09-04T19:43:45.424 INFO:tasks.workunit.client.0.smithi174.stdout: 18 16 1067 1051 0.228054 0.21875 0.28291 0.271154 2024-09-04T19:43:46.424 INFO:tasks.workunit.client.0.smithi174.stdout: 19 16 1124 1108 0.227768 0.222656 0.279865 0.272509 2024-09-04T19:43:47.424 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:43:47.424756+0000 min lat: 0.169645 max lat: 0.63559 avg lat: 0.273338 2024-09-04T19:43:47.424 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:43:47.425 INFO:tasks.workunit.client.0.smithi174.stdout: 20 16 1179 1163 0.22712 0.214844 0.278672 0.273338 2024-09-04T19:43:48.424 INFO:tasks.workunit.client.0.smithi174.stdout: 21 16 1237 1221 0.227092 0.226562 0.275818 0.273439 2024-09-04T19:43:49.424 INFO:tasks.workunit.client.0.smithi174.stdout: 22 16 1301 1285 0.228131 0.25 0.242938 0.272886 2024-09-04T19:43:50.425 INFO:tasks.workunit.client.0.smithi174.stdout: 23 16 1363 1347 0.228741 0.242188 0.283155 0.272036 2024-09-04T19:43:51.425 INFO:tasks.workunit.client.0.smithi174.stdout: 24 16 1419 1403 0.228323 0.21875 0.247469 0.271342 2024-09-04T19:43:52.425 INFO:tasks.workunit.client.0.smithi174.stdout: 25 16 1479 1463 0.228564 0.234375 0.302773 0.271617 2024-09-04T19:43:53.425 INFO:tasks.workunit.client.0.smithi174.stdout: 26 16 1543 1527 0.229387 0.25 0.242484 0.27143 2024-09-04T19:43:54.425 INFO:tasks.workunit.client.0.smithi174.stdout: 27 16 1591 1575 0.227835 0.1875 0.276621 0.271521 2024-09-04T19:43:55.425 INFO:tasks.workunit.client.0.smithi174.stdout: 28 16 1627 1611 0.224719 0.140625 0.268275 0.276411 2024-09-04T19:43:56.425 INFO:tasks.workunit.client.0.smithi174.stdout: 29 16 1687 1671 0.225051 0.234375 0.280127 0.276564 2024-09-04T19:43:57.425 INFO:tasks.workunit.client.0.smithi174.stdout: 30 16 1754 1738 0.226271 0.261719 0.244232 0.275336 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 30.2135 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 1754 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4096 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4096 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 0.226772 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 0.0337424 2024-09-04T19:43:58.426 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 0.261719 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 0.132812 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 58 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 8.63806 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 67 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 34 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 0.274945 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.0674976 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 0.777684 2024-09-04T19:43:58.427 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.169645 2024-09-04T19:43:58.547 INFO:tasks.workunit.client.0.smithi174.stdout:after bench 2024-09-04T19:43:58.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-04T19:43:58.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98286 2024-09-04T19:43:58.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-04T19:43:59.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98286 2024-09-04T19:43:59.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (98286) - No such process 2024-09-04T19:43:59.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-04T19:44:00.357 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:44:00.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 100840 2024-09-04T19:44:00.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-04T19:44:01.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 100840 2024-09-04T19:44:01.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (100840) - No such process 2024-09-04T19:44:01.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-04T19:44:01.816 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:44:01.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103369 2024-09-04T19:44:01.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-04T19:44:02.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103369 2024-09-04T19:44:02.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (103369) - No such process 2024-09-04T19:44:02.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-04T19:44:03.288 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:44:03.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 105923 2024-09-04T19:44:03.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-04T19:44:04.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 105923 2024-09-04T19:44:04.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (105923) - No such process 2024-09-04T19:44:04.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-04T19:44:04.848 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 is already down. 2024-09-04T19:44:04.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:44:07.487 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:07.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:44:09.879 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:44:12.279 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:12.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:44:14.671 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:14.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-04T19:44:14.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-04T19:44:14.732 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:15.921 INFO:tasks.workunit.client.0.smithi174.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-04T19:44:15.921 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-04T19:44:15.921 INFO:tasks.workunit.client.0.smithi174.stdout:2 : device size 0x100000000 : using 0x80536000(2.0 GiB) 2024-09-04T19:44:15.921 INFO:tasks.workunit.client.0.smithi174.stdout:Expanding DB/WAL... 2024-09-04T19:44:15.921 INFO:tasks.workunit.client.0.smithi174.stdout:0 : expanding to 0x536870912 2024-09-04T19:44:15.939 INFO:tasks.workunit.client.0.smithi174.stdout:0 : size label updated to 536870912 2024-09-04T19:44:15.939 INFO:tasks.workunit.client.0.smithi174.stdout:1 : expanding to 0x1073741824 2024-09-04T19:44:15.947 INFO:tasks.workunit.client.0.smithi174.stdout:1 : size label updated to 1073741824 2024-09-04T19:44:15.947 INFO:tasks.workunit.client.0.smithi174.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-04T19:44:16.089 INFO:tasks.workunit.client.0.smithi174.stdout:2 : size label updated to 4294967296 2024-09-04T19:44:21.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-04T19:44:21.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-04T19:44:21.475 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:22.652 INFO:tasks.workunit.client.0.smithi174.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-04T19:44:22.652 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-04T19:44:22.652 INFO:tasks.workunit.client.0.smithi174.stdout:2 : device size 0x101000000 : using 0x81580000(2.0 GiB) 2024-09-04T19:44:22.652 INFO:tasks.workunit.client.0.smithi174.stdout:Expanding DB/WAL... 2024-09-04T19:44:22.652 INFO:tasks.workunit.client.0.smithi174.stdout:0 : expanding to 0x536870912 2024-09-04T19:44:22.668 INFO:tasks.workunit.client.0.smithi174.stdout:0 : size label updated to 536870912 2024-09-04T19:44:22.668 INFO:tasks.workunit.client.0.smithi174.stdout:1 : expanding to 0x1073741824 2024-09-04T19:44:22.676 INFO:tasks.workunit.client.0.smithi174.stdout:1 : size label updated to 1073741824 2024-09-04T19:44:22.676 INFO:tasks.workunit.client.0.smithi174.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-04T19:44:22.835 INFO:tasks.workunit.client.0.smithi174.stdout:2 : size label updated to 4311744512 2024-09-04T19:44:28.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-04T19:44:28.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-04T19:44:28.320 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-04T19:44:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-04T19:44:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:2 : device size 0x100020000 : using 0x80599000(2.0 GiB) 2024-09-04T19:44:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:Expanding DB/WAL... 2024-09-04T19:44:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:0 : expanding to 0x536870912 2024-09-04T19:44:29.552 INFO:tasks.workunit.client.0.smithi174.stdout:0 : size label updated to 536870912 2024-09-04T19:44:29.552 INFO:tasks.workunit.client.0.smithi174.stdout:1 : expanding to 0x1073741824 2024-09-04T19:44:29.557 INFO:tasks.workunit.client.0.smithi174.stdout:1 : size label updated to 1073741824 2024-09-04T19:44:29.557 INFO:tasks.workunit.client.0.smithi174.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-04T19:44:29.765 INFO:tasks.workunit.client.0.smithi174.stdout:2 : size label updated to 4295098368 2024-09-04T19:44:35.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-04T19:44:35.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-04T19:44:35.397 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:36.603 INFO:tasks.workunit.client.0.smithi174.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-04T19:44:36.604 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-04T19:44:36.604 INFO:tasks.workunit.client.0.smithi174.stdout:2 : device size 0xfff00000 : using 0x80470000(2.0 GiB) 2024-09-04T19:44:36.604 INFO:tasks.workunit.client.0.smithi174.stdout:Expanding DB/WAL... 2024-09-04T19:44:36.604 INFO:tasks.workunit.client.0.smithi174.stdout:0 : expanding to 0x536870912 2024-09-04T19:44:36.617 INFO:tasks.workunit.client.0.smithi174.stdout:0 : size label updated to 536870912 2024-09-04T19:44:36.617 INFO:tasks.workunit.client.0.smithi174.stdout:1 : expanding to 0x1073741824 2024-09-04T19:44:36.625 INFO:tasks.workunit.client.0.smithi174.stdout:1 : size label updated to 1073741824 2024-09-04T19:44:36.625 INFO:tasks.workunit.client.0.smithi174.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-04T19:44:36.759 INFO:tasks.workunit.client.0.smithi174.stdout:2 : size label updated to 4293918720 2024-09-04T19:44:42.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:44:44.567 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:44.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:44:46.959 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:46.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:44:49.347 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:49.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:44:51.739 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:51.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-04T19:44:51.788 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:53.857 INFO:tasks.workunit.client.0.smithi174.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-04T19:44:53.857 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-04T19:44:53.857 INFO:tasks.workunit.client.0.smithi174.stdout:2 : device size 0x100000000 : using 0x536000(5.2 MiB) 2024-09-04T19:44:53.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-04T19:44:53.886 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:44:56.807 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-04T19:44:56.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:44:58.695 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:44:58.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-04T19:44:58.741 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:45:01.919 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-04T19:45:01.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:45:03.823 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:45:03.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-04T19:45:03.871 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:45:06.779 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-04T19:45:06.779 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-04T19:45:06.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:45:08.159 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:45:08.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-04T19:45:08.199 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:45:09.047 INFO:tasks.workunit.client.0.smithi174.stderr:Migrate to WAL device isn't supported. 2024-09-04T19:45:09.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-04T19:45:09.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:45:11.427 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:45:11.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-04T19:45:11.476 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:45:14.143 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-04T19:45:14.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:45:16.015 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:45:16.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-04T19:45:16.061 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:45:18.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:45:20.075 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:45:20.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:45:20.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:45:20.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:45:20.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:20.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:45:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:45:20.102 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:45:20.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:45:20.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:45:20.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-04T19:45:20.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:45:20.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:45:20.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:45:20.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:45:20.132 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:20.127+0000 7f7256c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:20.166 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:20.159+0000 7f7256c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:20.167 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:20.163+0000 7f7256c0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:20.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:45:20.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:45:20.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:45:20.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:20.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:45:21.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:21.666 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:21.659+0000 7f7256c0a7c0 -1 Falling back to public interface 2024-09-04T19:45:22.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:22.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:22.065 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:45:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:45:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:45:22.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:23.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:23.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:23.583 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:45:23.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:45:23.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:23.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:45:24.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:25.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:25.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:25.035 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:45:25.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:45:25.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:25.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:45:25.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:25.858 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:25.851+0000 7f7256c0a7c0 -1 osd.0 27 log_to_monitors true 2024-09-04T19:45:26.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:26.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:26.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:45:26.444 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:45:26.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:26.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:45:26.933 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/1062153952,v1:127.0.0.1:6803/1062153952] [v2:127.0.0.1:6804/1062153952,v1:127.0.0.1:6805/1062153952] exists,up 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:45:26.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:45:26.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:45:26.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:45:26.934 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=119435 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:45:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:45:26.939 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:45:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-04T19:45:26.941 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:45:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:45:26.969 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:26.963+0000 7f47a71077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:26.996 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:26.991+0000 7f47a71077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:26.991+0000 7f47a71077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:45:27.364 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:45:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:45:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:45:27.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:28.582 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:28.575+0000 7f47a71077c0 -1 Falling back to public interface 2024-09-04T19:45:28.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:28.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:28.760 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:45:28.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:45:28.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:28.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:45:29.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:30.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:30.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:30.167 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:45:30.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:45:30.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:30.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:45:30.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:31.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:31.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:31.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:45:31.589 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:45:31.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:31.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:45:31.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:32.556 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:32.551+0000 7f47a71077c0 -1 osd.1 30 log_to_monitors true 2024-09-04T19:45:32.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:32.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:32.996 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:45:32.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:45:32.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:32.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:45:33.570 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/2928133059,v1:127.0.0.1:6811/2928133059] [v2:127.0.0.1:6812/2928133059,v1:127.0.0.1:6813/2928133059] exists,up 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:45:33.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:45:33.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:45:33.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:45:33.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-04T19:45:33.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=120408 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-04T19:45:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:45:33.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:45:33.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:45:33.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-04T19:45:33.575 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:45:33.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:45:33.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:45:33.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-04T19:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:45:33.605 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:33.599+0000 7f7c0a6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:33.705 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:33.699+0000 7f7c0a6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:33.707 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:33.703+0000 7f7c0a6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:34.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:34.177 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:45:34.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:45:34.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:34.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:45:34.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:34.742 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:34.739+0000 7f7c0a6187c0 -1 Falling back to public interface 2024-09-04T19:45:35.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:35.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:35.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:45:35.628 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:45:35.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:35.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:45:36.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:45:37.050 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:45:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:45:37.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:37.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:38.055 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:38.051+0000 7f7c0a6187c0 -1 osd.2 33 log_to_monitors true 2024-09-04T19:45:38.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:38.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:38.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:45:38.451 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:45:38.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:38.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:45:39.243 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 45 up_thru 31 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/3533070053,v1:127.0.0.1:6819/3533070053] [v2:127.0.0.1:6820/3533070053,v1:127.0.0.1:6821/3533070053] exists,up 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:45:39.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:45:39.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:45:39.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:45:39.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=121398 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:45:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:45:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:45:39.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:45:39.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-04T19:45:39.252 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:45:39.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:45:39.281 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:39.279+0000 7fdc4c79c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:39.395 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:39.395+0000 7fdc4c79c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:39.490 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:39.395+0000 7fdc4c79c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:39.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:45:39.777 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:45:39.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:39.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:45:40.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:40.738 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:40.735+0000 7fdc4c79c7c0 -1 Falling back to public interface 2024-09-04T19:45:41.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:41.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:41.261 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:45:41.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:45:41.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:41.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:45:41.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:42.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:42.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:42.671 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:45:42.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:45:42.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:42.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:45:43.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:44.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:44.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:44.088 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:45:44.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:45:44.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:44.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:45:44.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:45:44.933 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:45:44.931+0000 7fdc4c79c7c0 -1 osd.3 35 log_to_monitors true 2024-09-04T19:45:45.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:45:45.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:45:45.519 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:45:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:45:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:45:45.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:45:46.225 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 49 up_thru 34 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/2977680702,v1:127.0.0.1:6827/2977680702] [v2:127.0.0.1:6828/2977680702,v1:127.0.0.1:6829/2977680702] exists,up 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:45:46.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:45:46.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:45:46.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:45:46.226 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-04T19:45:46.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=122343 2024-09-04T19:45:46.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-04T19:45:46.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:45:46.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:45:46.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:45:46.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:45:46.230 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:45:46.230 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:45:46.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:45:46.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:45:46.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:45:46.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:45:46.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:45:46.968 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:45:46.968 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-04T19:45:46.969 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-04T19:45:46.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:45:46.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:45:46.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:45:47.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757255 2024-09-04T19:45:47.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757255 2024-09-04T19:45:47.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757255' 2024-09-04T19:45:47.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:45:47.220 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:45:47.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659141 2024-09-04T19:45:47.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659141 2024-09-04T19:45:47.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2024-09-04T19:45:47.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:45:47.375 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:45:47.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528324 2024-09-04T19:45:47.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528324 2024-09-04T19:45:47.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324' 2024-09-04T19:45:47.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:45:47.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-04T19:45:47.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397507 2024-09-04T19:45:47.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397507 2024-09-04T19:45:47.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324 3-210453397507' 2024-09-04T19:45:47.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:45:47.683 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-163208757255 2024-09-04T19:45:47.683 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:45:47.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:45:47.685 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-163208757255 2024-09-04T19:45:47.685 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:45:47.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757255 2024-09-04T19:45:47.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2024-09-04T19:45:47.686 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 163208757255 2024-09-04T19:45:47.686 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:45:48.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757255 -lt 163208757255 2024-09-04T19:45:48.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:45:48.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659141 2024-09-04T19:45:48.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:45:48.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:45:48.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659141 2024-09-04T19:45:48.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:45:48.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659141 2024-09-04T19:45:48.146 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 176093659141 2024-09-04T19:45:48.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-09-04T19:45:48.147 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:45:48.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659141 2024-09-04T19:45:48.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:45:48.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-193273528324 2024-09-04T19:45:48.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:45:48.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:45:48.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-193273528324 2024-09-04T19:45:48.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:45:48.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528324 2024-09-04T19:45:48.591 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 193273528324 2024-09-04T19:45:48.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2024-09-04T19:45:48.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:45:49.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528324 -lt 193273528324 2024-09-04T19:45:49.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:45:49.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397507 2024-09-04T19:45:49.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:45:49.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-04T19:45:49.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397507 2024-09-04T19:45:49.007 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:45:49.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397507 2024-09-04T19:45:49.008 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 210453397507 2024-09-04T19:45:49.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-09-04T19:45:49.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-04T19:45:49.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397507 2024-09-04T19:45:49.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:45:50.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:45:50.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-04T19:45:50.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-04T19:45:50.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:45:50.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:45:50.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:45:51.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-04T19:45:51.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:45:51.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:45:51.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:45:51.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:45:51.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:45:51.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:45:51.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:45:51.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-04T19:45:51.822 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:45:51.823 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:45:51.823 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:45:52.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-04T19:45:52.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:45:52.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:45:52.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-04T19:45:52.402 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:45:52.402 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-04T19:45:52.402 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_123667 2024-09-04T19:45:52.402 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:45:52.402 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:45:53.402 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 53 37 0.144522 0.144531 0.167882 0.381136 2024-09-04T19:45:54.402 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 106 90 0.175762 0.207031 0.233949 0.318815 2024-09-04T19:45:55.402 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 165 149 0.193985 0.230469 0.175428 0.306331 2024-09-04T19:45:56.403 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 226 210 0.205049 0.238281 0.26457 0.294541 2024-09-04T19:45:57.403 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 285 269 0.210126 0.230469 0.28506 0.289638 2024-09-04T19:45:58.403 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 336 320 0.208304 0.199219 0.318168 0.293114 2024-09-04T19:45:59.403 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 396 380 0.212025 0.234375 0.251018 0.290502 2024-09-04T19:46:00.403 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 452 436 0.212863 0.21875 0.210318 0.286909 2024-09-04T19:46:01.403 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 515 499 0.216552 0.246094 0.168403 0.284469 2024-09-04T19:46:02.403 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 574 558 0.217941 0.230469 0.165424 0.284763 2024-09-04T19:46:03.403 INFO:tasks.workunit.client.0.smithi174.stdout: 11 16 632 616 0.218723 0.226562 0.193589 0.281436 2024-09-04T19:46:04.404 INFO:tasks.workunit.client.0.smithi174.stdout: 12 16 694 678 0.220676 0.242188 0.252134 0.279792 2024-09-04T19:46:05.403 INFO:tasks.workunit.client.0.smithi174.stdout: 13 16 761 745 0.223831 0.261719 0.166305 0.277388 2024-09-04T19:46:06.404 INFO:tasks.workunit.client.0.smithi174.stdout: 14 16 815 799 0.222908 0.210938 0.324348 0.276677 2024-09-04T19:46:07.404 INFO:tasks.workunit.client.0.smithi174.stdout: 15 16 881 865 0.225232 0.257812 0.172386 0.275431 2024-09-04T19:46:08.404 INFO:tasks.workunit.client.0.smithi174.stdout: 16 16 932 916 0.223605 0.199219 0.408879 0.278447 2024-09-04T19:46:09.404 INFO:tasks.workunit.client.0.smithi174.stdout: 17 16 988 972 0.223318 0.21875 0.0836047 0.277442 2024-09-04T19:46:10.404 INFO:tasks.workunit.client.0.smithi174.stdout: 18 16 1049 1033 0.224147 0.238281 0.319565 0.277218 2024-09-04T19:46:11.405 INFO:tasks.workunit.client.0.smithi174.stdout: 19 16 1111 1095 0.225094 0.242188 0.133335 0.275856 2024-09-04T19:46:12.405 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:46:12.405399+0000 min lat: 0.0836047 max lat: 0.573933 avg lat: 0.274098 2024-09-04T19:46:12.405 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:46:12.405 INFO:tasks.workunit.client.0.smithi174.stdout: 20 16 1175 1159 0.226338 0.25 0.27022 0.274098 2024-09-04T19:46:13.405 INFO:tasks.workunit.client.0.smithi174.stdout: 21 16 1228 1212 0.225417 0.207031 0.173251 0.272429 2024-09-04T19:46:14.405 INFO:tasks.workunit.client.0.smithi174.stdout: 22 16 1295 1279 0.227065 0.261719 0.197377 0.27363 2024-09-04T19:46:15.405 INFO:tasks.workunit.client.0.smithi174.stdout: 23 16 1362 1346 0.22857 0.261719 0.216632 0.272218 2024-09-04T19:46:16.405 INFO:tasks.workunit.client.0.smithi174.stdout: 24 16 1433 1417 0.230601 0.277344 0.215885 0.270444 2024-09-04T19:46:17.405 INFO:tasks.workunit.client.0.smithi174.stdout: 25 16 1498 1482 0.231532 0.253906 0.226039 0.268657 2024-09-04T19:46:18.405 INFO:tasks.workunit.client.0.smithi174.stdout: 26 16 1563 1547 0.232391 0.253906 0.153433 0.267821 2024-09-04T19:46:19.406 INFO:tasks.workunit.client.0.smithi174.stdout: 27 16 1626 1610 0.232897 0.246094 0.19008 0.266881 2024-09-04T19:46:20.406 INFO:tasks.workunit.client.0.smithi174.stdout: 28 16 1697 1681 0.234483 0.277344 0.260631 0.265267 2024-09-04T19:46:21.406 INFO:tasks.workunit.client.0.smithi174.stdout: 29 16 1771 1755 0.236364 0.289062 0.228554 0.263486 2024-09-04T19:46:22.406 INFO:tasks.workunit.client.0.smithi174.stdout: 30 16 1830 1814 0.236166 0.230469 0.208069 0.263351 2024-09-04T19:46:23.406 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 30.1708 2024-09-04T19:46:23.406 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 1830 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4096 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4096 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 0.236933 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 0.0287733 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 0.289062 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 0.144531 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 60 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 7.36597 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 74 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 37 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 0.263244 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.0590845 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 0.573933 2024-09-04T19:46:23.407 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.0836047 2024-09-04T19:46:23.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119435 2024-09-04T19:46:23.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-04T19:46:24.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119435 2024-09-04T19:46:24.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (119435) - No such process 2024-09-04T19:46:24.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-04T19:46:24.927 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:46:24.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120408 2024-09-04T19:46:24.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-04T19:46:25.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120408 2024-09-04T19:46:25.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (120408) - No such process 2024-09-04T19:46:25.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-04T19:46:26.491 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:46:26.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121398 2024-09-04T19:46:26.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-04T19:46:27.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121398 2024-09-04T19:46:27.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (121398) - No such process 2024-09-04T19:46:27.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-04T19:46:27.971 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:46:27.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122343 2024-09-04T19:46:27.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-04T19:46:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122343 2024-09-04T19:46:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (122343) - No such process 2024-09-04T19:46:28.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-04T19:46:29.531 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 is already down. 2024-09-04T19:46:29.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:46:31.431 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:46:31.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-04T19:46:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:512+0 records in 2024-09-04T19:46:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:512+0 records out 2024-09-04T19:46:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.303857 s, 1.8 GB/s 2024-09-04T19:46:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-04T19:46:31.780 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:46:37.428 INFO:tasks.workunit.client.0.smithi174.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-04T19:46:37.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:46:39.799 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:46:39.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:46:41.687 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:46:41.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-04T19:46:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:46:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:46:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.562262 s, 1.9 GB/s 2024-09-04T19:46:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-04T19:46:42.290 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:46:50.423 INFO:tasks.workunit.client.0.smithi174.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-04T19:46:50.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-04T19:46:50.455 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:46:53.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:46:55.747 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:46:55.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:46:57.143 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:46:57.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-04T19:46:57.183 INFO:tasks.workunit.client.0.smithi174.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-04T19:46:57.183 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:46:58.543 INFO:tasks.workunit.client.0.smithi174.stdout:DB device added 2024-09-04T19:46:58.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-04T19:46:58.574 INFO:tasks.workunit.client.0.smithi174.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-04T19:46:58.575 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:47:00.695 INFO:tasks.workunit.client.0.smithi174.stdout:WAL device added 2024-09-04T19:47:00.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-04T19:47:00.729 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:47:03.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:47:06.283 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:47:06.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:47:08.175 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:47:08.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-04T19:47:08.218 INFO:tasks.workunit.client.0.smithi174.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-04T19:47:08.218 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:47:10.347 INFO:tasks.workunit.client.0.smithi174.stdout:WAL device added 2024-09-04T19:47:10.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-04T19:47:10.379 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:47:13.531 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-04T19:47:13.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:47:15.425 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:47:15.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:47:15.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:47:15.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:47:15.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:47:15.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:47:15.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:47:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:47:15.446 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:47:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:47:15.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:47:15.447 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-04T19:47:15.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:47:15.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:47:15.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:47:15.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:47:15.483 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:15.482+0000 7f25f1d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:15.485 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:15.482+0000 7f25f1d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:15.487 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:15.486+0000 7f25f1d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:15.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:47:15.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:47:15.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:47:15.883 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:16.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:17.262 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:17.258+0000 7f25f1d8a7c0 -1 Falling back to public interface 2024-09-04T19:47:17.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:17.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:17.353 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:47:17.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:47:17.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:17.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:17.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:47:18.749 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:47:18.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:18.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:19.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:20.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:20.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:20.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:47:20.163 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:47:20.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:20.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:20.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:21.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:21.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:21.563 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:47:21.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:47:21.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:21.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:21.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:22.230 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:22.226+0000 7f25f1d8a7c0 -1 osd.0 52 log_to_monitors true 2024-09-04T19:47:22.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:22.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:22.982 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:47:22.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:47:22.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:22.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:47:23.433 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 63 up_thru 63 down_at 53 last_clean_interval [38,52) [v2:127.0.0.1:6802/1800217885,v1:127.0.0.1:6803/1800217885] [v2:127.0.0.1:6804/1800217885,v1:127.0.0.1:6805/1800217885] exists,up 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:47:23.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:47:23.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:47:23.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:47:23.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:47:23.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=129818 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:47:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:47:23.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:23.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-04T19:47:23.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:47:23.443 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:47:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:47:23.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-04T19:47:23.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:47:23.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:47:23.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:47:23.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:47:23.465 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:23.462+0000 7f848feae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:23.488 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:23.486+0000 7f848feae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:23.488 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:23.486+0000 7f848feae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:23.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:47:23.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:47:23.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:47:23.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:23.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:47:24.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:24.966 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:24.962+0000 7f848feae7c0 -1 Falling back to public interface 2024-09-04T19:47:25.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:25.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:25.335 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:47:25.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:47:25.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:25.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:47:25.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:47:27.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:28.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:28.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:28.168 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:47:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:47:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:47:28.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:29.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:29.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:29.572 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:47:29.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:47:29.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:29.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:47:29.625 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:29.622+0000 7f848feae7c0 -1 osd.1 55 log_to_monitors true 2024-09-04T19:47:30.097 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 66 up_thru 53 down_at 56 last_clean_interval [41,55) [v2:127.0.0.1:6810/1776181123,v1:127.0.0.1:6811/1776181123] [v2:127.0.0.1:6812/1776181123,v1:127.0.0.1:6813/1776181123] exists,up 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:47:30.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:47:30.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:47:30.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:47:30.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-04T19:47:30.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=130821 2024-09-04T19:47:30.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:47:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:47:30.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:47:30.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:47:30.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:47:30.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:47:30.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:47:30.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:47:30.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:47:30.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-04T19:47:30.106 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:47:30.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:47:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:30.134+0000 7fbf362fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:30.141 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:30.138+0000 7fbf362fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:30.142 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:30.138+0000 7fbf362fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:30.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:47:30.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:30.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:31.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:32.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:32.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:32.151 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:47:32.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:47:32.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:32.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:32.241 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:32.237+0000 7fbf362fb7c0 -1 Falling back to public interface 2024-09-04T19:47:32.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:33.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:33.575 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:47:33.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:47:33.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:33.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:35.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:35.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:35.041 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:47:35.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:47:35.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:35.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:35.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:36.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:36.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:36.447 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:47:36.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:47:36.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:36.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:36.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:37.754 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:37.749+0000 7fbf362fb7c0 -1 osd.2 58 log_to_monitors true 2024-09-04T19:47:37.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:37.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:37.884 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:47:37.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:47:37.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:37.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:47:38.626 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 70 up_thru 56 down_at 59 last_clean_interval [45,58) [v2:127.0.0.1:6818/3442872008,v1:127.0.0.1:6819/3442872008] [v2:127.0.0.1:6820/3442872008,v1:127.0.0.1:6821/3442872008] exists,up 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:47:38.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:47:38.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:47:38.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:47:38.627 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-04T19:47:38.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=131799 2024-09-04T19:47:38.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:47:38.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:47:38.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:47:38.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:47:38.632 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:47:38.632 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:47:38.632 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:47:38.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:38.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:47:38.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-04T19:47:38.634 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:47:38.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-04T19:47:38.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:47:38.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-04T19:47:38.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-04T19:47:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:47:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:47:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:47:38.664 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:38.661+0000 7f7e5c36c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:38.749 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:38.745+0000 7f7e5c36c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:38.770 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:38.765+0000 7f7e5c36c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:47:39.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-04T19:47:39.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:47:39.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:47:39.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:47:39.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:47:39.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:39.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:47:39.237 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:47:39.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:39.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:47:39.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:40.334 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:40.329+0000 7f7e5c36c7c0 -1 Falling back to public interface 2024-09-04T19:47:40.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:40.949 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:47:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:47:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:40.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:47:41.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:42.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:42.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:42.379 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:47:42.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:47:42.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:42.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:47:42.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:43.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:43.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:43.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:47:43.793 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:47:43.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:43.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:47:44.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:47:44.895 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:47:44.889+0000 7f7e5c36c7c0 -1 osd.3 60 log_to_monitors true 2024-09-04T19:47:45.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:47:45.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:47:45.215 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:47:45.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:47:45.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:47:45.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:47:45.719 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 75 up_thru 59 down_at 61 last_clean_interval [49,60) [v2:127.0.0.1:6826/2098861825,v1:127.0.0.1:6827/2098861825] [v2:127.0.0.1:6828/2098861825,v1:127.0.0.1:6829/2098861825] exists,up 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:47:45.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:47:45.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:47:45.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:47:45.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-04T19:47:45.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=132803 2024-09-04T19:47:45.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-04T19:47:45.832 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:47:45.832 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-04T19:47:45.832 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_133760 2024-09-04T19:47:45.832 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:47:45.832 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:47:46.832 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 23 7 0.02734 0.0273438 0.266169 0.201406 2024-09-04T19:47:47.832 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 69 53 0.103501 0.179688 0.269999 0.560756 2024-09-04T19:47:48.833 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 119 103 0.134096 0.195312 0.267967 0.4492 2024-09-04T19:47:49.833 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 173 157 0.1533 0.210938 0.32675 0.394058 2024-09-04T19:47:50.833 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 222 206 0.160916 0.191406 0.30089 0.373472 2024-09-04T19:47:51.833 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 275 259 0.168597 0.207031 0.18372 0.356954 2024-09-04T19:47:52.833 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 332 316 0.176316 0.222656 0.300547 0.348653 2024-09-04T19:47:53.833 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 385 369 0.180153 0.207031 0.359134 0.340806 2024-09-04T19:47:54.833 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 438 422 0.183137 0.207031 0.267615 0.335683 2024-09-04T19:47:55.833 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 497 481 0.187867 0.230469 0.283275 0.330416 2024-09-04T19:47:56.834 INFO:tasks.workunit.client.0.smithi174.stdout: 11 16 540 524 0.186056 0.167969 0.276208 0.326312 2024-09-04T19:47:57.834 INFO:tasks.workunit.client.0.smithi174.stdout: 12 16 594 578 0.188127 0.210938 0.300176 0.328899 2024-09-04T19:47:58.834 INFO:tasks.workunit.client.0.smithi174.stdout: 13 16 655 639 0.191982 0.238281 0.284324 0.323271 2024-09-04T19:47:59.834 INFO:tasks.workunit.client.0.smithi174.stdout: 14 16 706 690 0.192497 0.199219 0.284902 0.320203 2024-09-04T19:48:00.834 INFO:tasks.workunit.client.0.smithi174.stdout: 15 16 758 742 0.193204 0.203125 0.333462 0.319817 2024-09-04T19:48:01.834 INFO:tasks.workunit.client.0.smithi174.stdout: 16 16 814 798 0.194798 0.21875 0.291743 0.317479 2024-09-04T19:48:02.834 INFO:tasks.workunit.client.0.smithi174.stdout: 17 16 862 846 0.194368 0.1875 0.300272 0.31885 2024-09-04T19:48:03.835 INFO:tasks.workunit.client.0.smithi174.stdout: 18 16 918 902 0.19572 0.21875 0.318063 0.316148 2024-09-04T19:48:04.835 INFO:tasks.workunit.client.0.smithi174.stdout: 19 16 974 958 0.196931 0.21875 0.181807 0.315294 2024-09-04T19:48:05.835 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:48:05.835654+0000 min lat: 0.154067 max lat: 1.40342 avg lat: 0.313252 2024-09-04T19:48:05.835 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:48:05.835 INFO:tasks.workunit.client.0.smithi174.stdout: 20 16 1029 1013 0.197825 0.214844 0.334919 0.313252 2024-09-04T19:48:06.835 INFO:tasks.workunit.client.0.smithi174.stdout: 21 16 1085 1069 0.19882 0.21875 0.182532 0.312432 2024-09-04T19:48:07.835 INFO:tasks.workunit.client.0.smithi174.stdout: 22 16 1140 1124 0.199548 0.214844 0.305044 0.310978 2024-09-04T19:48:08.835 INFO:tasks.workunit.client.0.smithi174.stdout: 23 16 1190 1174 0.199363 0.195312 0.32513 0.311111 2024-09-04T19:48:09.835 INFO:tasks.workunit.client.0.smithi174.stdout: 24 16 1247 1231 0.200332 0.222656 0.320077 0.31044 2024-09-04T19:48:10.836 INFO:tasks.workunit.client.0.smithi174.stdout: 25 16 1302 1286 0.200909 0.214844 0.317721 0.310639 2024-09-04T19:48:11.836 INFO:tasks.workunit.client.0.smithi174.stdout: 26 16 1351 1335 0.200543 0.191406 0.299622 0.309896 2024-09-04T19:48:12.836 INFO:tasks.workunit.client.0.smithi174.stdout: 27 16 1394 1378 0.199336 0.167969 0.216166 0.311983 2024-09-04T19:48:13.836 INFO:tasks.workunit.client.0.smithi174.stdout: 28 16 1444 1428 0.199191 0.195312 0.198141 0.311343 2024-09-04T19:48:14.836 INFO:tasks.workunit.client.0.smithi174.stdout: 29 16 1499 1483 0.19973 0.214844 0.302053 0.311263 2024-09-04T19:48:15.836 INFO:tasks.workunit.client.0.smithi174.stdout: 30 16 1554 1538 0.200233 0.214844 0.309272 0.310217 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 30.1432 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 1554 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4096 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4096 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 0.201382 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 0.0366573 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 0.238281 2024-09-04T19:48:16.837 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 0.0273438 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 51 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 9.38426 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 61 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 7 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 0.309756 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.114104 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 1.40342 2024-09-04T19:48:16.838 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.154067 2024-09-04T19:48:16.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 129818 2024-09-04T19:48:16.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-04T19:48:17.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 129818 2024-09-04T19:48:17.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (129818) - No such process 2024-09-04T19:48:17.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-04T19:48:18.590 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:48:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 130821 2024-09-04T19:48:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-04T19:48:19.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 130821 2024-09-04T19:48:19.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (130821) - No such process 2024-09-04T19:48:19.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-04T19:48:20.083 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already down. 2024-09-04T19:48:20.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 131799 2024-09-04T19:48:20.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-04T19:48:21.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 131799 2024-09-04T19:48:21.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (131799) - No such process 2024-09-04T19:48:21.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-04T19:48:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-04T19:48:21.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 132803 2024-09-04T19:48:21.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-04T19:48:22.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 132803 2024-09-04T19:48:22.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (132803) - No such process 2024-09-04T19:48:22.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-04T19:48:23.129 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 is already down. 2024-09-04T19:48:23.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:48:25.515 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:48:25.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-04T19:48:26.106 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:48:26.106 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:48:26.106 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.56571 s, 1.9 GB/s 2024-09-04T19:48:26.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-04T19:48:26.131 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:48:35.791 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-04T19:48:35.791 INFO:tasks.workunit.client.0.smithi174.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-04T19:48:35.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:48:38.179 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:48:38.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-04T19:48:38.501 INFO:tasks.workunit.client.0.smithi174.stderr:512+0 records in 2024-09-04T19:48:38.501 INFO:tasks.workunit.client.0.smithi174.stderr:512+0 records out 2024-09-04T19:48:38.501 INFO:tasks.workunit.client.0.smithi174.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.301377 s, 1.8 GB/s 2024-09-04T19:48:38.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-04T19:48:38.525 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:48:44.447 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-04T19:48:44.447 INFO:tasks.workunit.client.0.smithi174.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-04T19:48:44.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-04T19:48:44.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-04T19:48:46.903 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:48:46.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:48:49.315 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:48:49.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-04T19:48:49.905 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:48:49.905 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:48:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.564127 s, 1.9 GB/s 2024-09-04T19:48:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-04T19:48:49.929 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:48:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-04T19:48:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-04T19:48:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-04T19:48:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-04T19:48:59.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:49:01.595 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:01.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-04T19:49:01.638 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:49:03.767 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-04T19:49:03.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-04T19:49:03.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-04T19:49:05.283 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:05.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-04T19:49:07.383 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:07.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-04T19:49:07.985 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:49:07.985 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:49:07.985 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.582353 s, 1.8 GB/s 2024-09-04T19:49:07.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-04T19:49:08.009 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:49:08.871 INFO:tasks.workunit.client.0.smithi174.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-04T19:49:08.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-04T19:49:08.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:49:11.251 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:11.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-04T19:49:18.546 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:49:18.546 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:49:18.546 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.13965 s, 150 MB/s 2024-09-04T19:49:18.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-04T19:49:18.572 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:49:21.727 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-04T19:49:21.727 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-04T19:49:21.727 INFO:tasks.workunit.client.0.smithi174.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-04T19:49:21.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-04T19:49:23.603 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:23.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-04T19:49:24.183 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records in 2024-09-04T19:49:24.183 INFO:tasks.workunit.client.0.smithi174.stderr:1024+0 records out 2024-09-04T19:49:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.562562 s, 1.9 GB/s 2024-09-04T19:49:24.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-04T19:49:24.207 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:49:33.331 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-04T19:49:33.331 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-04T19:49:33.331 INFO:tasks.workunit.client.0.smithi174.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-04T19:49:33.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-04T19:49:35.215 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:49:35.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:49:35.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:49:35.235 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:49:35.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:49:35.238 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:49:35.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:49:35.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:49:35.238 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-04T19:49:35.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:49:35.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:49:35.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:49:35.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:49:35.270 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:35.264+0000 7f5faf4877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:35.273 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:35.268+0000 7f5faf4877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:35.275 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:35.272+0000 7f5faf4877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:35.705 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:49:35.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:49:35.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:35.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:36.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:36.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:36.860+0000 7f5faf4877c0 -1 Falling back to public interface 2024-09-04T19:49:37.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:37.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:37.155 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:49:37.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:49:37.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:37.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:37.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:38.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:38.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:38.571 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:49:38.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:49:38.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:38.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:38.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:39.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:39.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:39.984 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:49:39.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:49:39.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:39.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:40.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:41.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:41.397 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:49:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:49:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:41.794 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:41.788+0000 7f5faf4877c0 -1 osd.0 76 log_to_monitors true 2024-09-04T19:49:41.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:42.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:42.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:42.800 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:49:42.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:49:42.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:42.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:49:43.256 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 87 up_thru 87 down_at 77 last_clean_interval [63,76) [v2:127.0.0.1:6802/3635994666,v1:127.0.0.1:6803/3635994666] [v2:127.0.0.1:6804/3635994666,v1:127.0.0.1:6805/3635994666] exists,up 256dd40b-59b2-408c-b6b4-c2218bb84138 2024-09-04T19:49:43.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:49:43.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:49:43.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:49:43.257 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=139934 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:49:43.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:49:43.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:49:43.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-04T19:49:43.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-04T19:49:43.264 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:49:43.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:49:43.286 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:43.284+0000 7feeed9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:43.289 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:43.284+0000 7feeed9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:43.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:43.284+0000 7feeed9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:49:43.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:43.695 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:49:43.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:49:43.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:43.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:49:44.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:44.534 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:44.528+0000 7feeed9187c0 -1 Falling back to public interface 2024-09-04T19:49:45.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:45.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:45.154 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:49:45.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:49:45.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:45.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:49:45.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:46.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:46.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:46.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:49:46.575 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:49:46.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:46.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:49:47.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:47.933 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:47.928+0000 7feeed9187c0 -1 osd.1 79 log_to_monitors true 2024-09-04T19:49:48.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:48.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:48.037 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:49:48.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:49:48.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:48.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-04T19:49:48.467 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 90 up_thru 77 down_at 80 last_clean_interval [66,79) [v2:127.0.0.1:6810/639807222,v1:127.0.0.1:6811/639807222] [v2:127.0.0.1:6812/639807222,v1:127.0.0.1:6813/639807222] exists,up 30112775-75b5-463d-83a5-bf8aa20d94aa 2024-09-04T19:49:48.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:49:48.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:49:48.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:49:48.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-04T19:49:48.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=140937 2024-09-04T19:49:48.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-04T19:49:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:49:48.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:49:48.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:49:48.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:49:48.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:49:48.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:49:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-04T19:49:48.473 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:49:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:49:48.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:49:48.501 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:48.496+0000 7f9d9ac7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:48.504 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:48.500+0000 7f9d9ac7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:48.505 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:48.500+0000 7f9d9ac7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:49.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-04T19:49:49.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:49:49.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-04T19:49:49.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:49:49.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:49:49.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:49.111 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:49:49.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:49:49.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:49.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:49:49.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:50.069 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:50.064+0000 7f9d9ac7e7c0 -1 Falling back to public interface 2024-09-04T19:49:50.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:50.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:50.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:49:50.544 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:49:50.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:50.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:49:50.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:51.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:51.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:51.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:49:51.953 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:49:51.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:51.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:49:52.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:53.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:53.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:53.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:49:53.453 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:49:53.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:53.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:49:53.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:54.428 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:54.424+0000 7f9d9ac7e7c0 -1 osd.2 82 log_to_monitors true 2024-09-04T19:49:54.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:54.887 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:49:54.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:54.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:49:54.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:54.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-04T19:49:55.581 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 93 up_thru 93 down_at 83 last_clean_interval [70,82) [v2:127.0.0.1:6818/1580742557,v1:127.0.0.1:6819/1580742557] [v2:127.0.0.1:6820/1580742557,v1:127.0.0.1:6821/1580742557] exists,up 4bc82cc6-fbc9-4666-ac6f-60e2b7f8773b 2024-09-04T19:49:55.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:49:55.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:49:55.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:49:55.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=141880 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:49:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:49:55.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:49:55.586 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:49:55.586 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:49:55.586 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:49:55.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:49:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:49:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:49:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-04T19:49:55.588 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-04T19:49:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:49:55.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-04T19:49:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-04T19:49:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:49:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:49:55.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:49:55.621 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:55.616+0000 7fbfa4a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:55.623 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:55.620+0000 7fbfa4a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:55.620+0000 7fbfa4a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:49:56.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-04T19:49:56.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:49:56.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-04T19:49:56.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:49:56.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:49:56.117 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:49:56.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:56.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:49:56.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:56.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:49:56.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:57.101 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:49:57.096+0000 7fbfa4a3f7c0 -1 Falling back to public interface 2024-09-04T19:49:57.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:57.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:57.563 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:49:57.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:49:57.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:57.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:49:58.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:49:59.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:49:59.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:49:59.018 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:49:59.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:49:59.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:49:59.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:49:59.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:00.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:00.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:00.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:50:00.429 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:50:00.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:00.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:50:00.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:01.384 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:01.380+0000 7fbfa4a3f7c0 -1 osd.3 84 log_to_monitors true 2024-09-04T19:50:01.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:01.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:01.891 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:50:01.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:50:01.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:01.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-04T19:50:02.402 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 97 up_thru 97 down_at 85 last_clean_interval [75,84) [v2:127.0.0.1:6826/2435255383,v1:127.0.0.1:6827/2435255383] [v2:127.0.0.1:6828/2435255383,v1:127.0.0.1:6829/2435255383] exists,up 59d144bb-00ee-453e-9caf-6b562763e52c 2024-09-04T19:50:02.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:50:02.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:50:02.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:50:02.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-04T19:50:02.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=142853 2024-09-04T19:50:02.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-04T19:50:02.434 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:50:02.435 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-04T19:50:02.435 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_143811 2024-09-04T19:50:02.435 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:50:02.435 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:50:03.434 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 60 44 0.171855 0.171875 0.124579 0.319925 2024-09-04T19:50:04.435 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 157 141 0.27536 0.378906 0.166619 0.219412 2024-09-04T19:50:05.435 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 247 231 0.300743 0.351562 0.143359 0.202229 2024-09-04T19:50:06.435 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 333 317 0.309528 0.335938 0.192279 0.195557 2024-09-04T19:50:07.435 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 455 439 0.342925 0.476562 0.109824 0.180259 2024-09-04T19:50:08.435 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 552 536 0.348915 0.378906 0.12509 0.1775 2024-09-04T19:50:09.435 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 654 638 0.355984 0.398438 0.158647 0.173237 2024-09-04T19:50:10.435 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 753 737 0.359821 0.386719 0.158979 0.172222 2024-09-04T19:50:11.435 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 864 848 0.368012 0.433594 0.124928 0.168722 2024-09-04T19:50:12.435 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 978 962 0.375738 0.445312 0.129926 0.165286 2024-09-04T19:50:13.436 INFO:tasks.workunit.client.0.smithi174.stdout: 11 16 1089 1073 0.380992 0.433594 0.134198 0.162459 2024-09-04T19:50:14.436 INFO:tasks.workunit.client.0.smithi174.stdout: 12 16 1200 1184 0.38537 0.433594 0.164558 0.160661 2024-09-04T19:50:15.436 INFO:tasks.workunit.client.0.smithi174.stdout: 13 16 1321 1305 0.392081 0.472656 0.116623 0.158827 2024-09-04T19:50:16.436 INFO:tasks.workunit.client.0.smithi174.stdout: 14 16 1415 1399 0.390298 0.367188 0.245402 0.157399 2024-09-04T19:50:17.436 INFO:tasks.workunit.client.0.smithi174.stdout: 15 16 1528 1512 0.393702 0.441406 0.100021 0.158119 2024-09-04T19:50:18.436 INFO:tasks.workunit.client.0.smithi174.stdout: 16 16 1634 1618 0.394971 0.414062 0.11861 0.157287 2024-09-04T19:50:19.436 INFO:tasks.workunit.client.0.smithi174.stdout: 17 16 1744 1728 0.397011 0.429688 0.15046 0.156618 2024-09-04T19:50:20.437 INFO:tasks.workunit.client.0.smithi174.stdout: 18 16 1856 1840 0.399257 0.4375 0.149067 0.155682 2024-09-04T19:50:21.437 INFO:tasks.workunit.client.0.smithi174.stdout: 19 16 1978 1962 0.403323 0.476562 0.112523 0.154413 2024-09-04T19:50:22.437 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:50:22.437888+0000 min lat: 0.0817916 max lat: 0.878738 avg lat: 0.155005 2024-09-04T19:50:22.437 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:50:22.437 INFO:tasks.workunit.client.0.smithi174.stdout: 20 16 2069 2053 0.400928 0.355469 0.175208 0.155005 2024-09-04T19:50:23.437 INFO:tasks.workunit.client.0.smithi174.stdout: 21 16 2188 2172 0.403969 0.464844 0.15876 0.154265 2024-09-04T19:50:24.437 INFO:tasks.workunit.client.0.smithi174.stdout: 22 16 2280 2264 0.401939 0.359375 0.195739 0.154325 2024-09-04T19:50:25.437 INFO:tasks.workunit.client.0.smithi174.stdout: 23 16 2394 2378 0.403823 0.445312 0.124972 0.154172 2024-09-04T19:50:26.437 INFO:tasks.workunit.client.0.smithi174.stdout: 24 16 2501 2485 0.40441 0.417969 0.199312 0.153906 2024-09-04T19:50:27.437 INFO:tasks.workunit.client.0.smithi174.stdout: 25 16 2581 2565 0.400732 0.3125 0.112723 0.155527 2024-09-04T19:50:28.437 INFO:tasks.workunit.client.0.smithi174.stdout: 26 16 2693 2677 0.402145 0.4375 0.178657 0.155012 2024-09-04T19:50:29.438 INFO:tasks.workunit.client.0.smithi174.stdout: 27 16 2805 2789 0.403452 0.4375 0.116447 0.154348 2024-09-04T19:50:30.438 INFO:tasks.workunit.client.0.smithi174.stdout: 28 16 2912 2896 0.403969 0.417969 0.157778 0.154276 2024-09-04T19:50:31.438 INFO:tasks.workunit.client.0.smithi174.stdout: 29 16 3029 3013 0.405796 0.457031 0.120766 0.153529 2024-09-04T19:50:32.438 INFO:tasks.workunit.client.0.smithi174.stdout: 30 16 3139 3123 0.406591 0.429688 0.18547 0.153504 2024-09-04T19:50:33.438 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 30.1571 2024-09-04T19:50:33.438 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 3139 2024-09-04T19:50:33.438 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4096 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4096 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 0.406595 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 0.0618939 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 0.476562 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 0.171875 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 104 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 15.8448 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 122 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 44 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 0.153517 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.0542999 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 0.878738 2024-09-04T19:50:33.439 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.0817916 2024-09-04T19:50:33.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-04T19:50:33.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:50:33.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:50:33.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:50:33.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:50:33.496 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:50:33.497 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:50:33.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:50:33.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:50:33.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:50:33.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:50:33.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:50:33.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:50:33.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:50:33.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:50:33.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:50:34.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-04T19:50:34.092 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-04T19:50:34.092 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-04T19:50:34.092 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-04T19:50:34.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:50:34.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:50:34.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:50:34.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154765 2024-09-04T19:50:34.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154765 2024-09-04T19:50:34.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-373662154765' 2024-09-04T19:50:34.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:50:34.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-04T19:50:34.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056652 2024-09-04T19:50:34.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056652 2024-09-04T19:50:34.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-373662154765 1-386547056652' 2024-09-04T19:50:34.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:50:34.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-04T19:50:34.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958538 2024-09-04T19:50:34.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958538 2024-09-04T19:50:34.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-373662154765 1-386547056652 2-399431958538' 2024-09-04T19:50:34.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:50:34.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-04T19:50:34.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827721 2024-09-04T19:50:34.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827721 2024-09-04T19:50:34.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-373662154765 1-386547056652 2-399431958538 3-416611827721' 2024-09-04T19:50:34.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:50:34.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-373662154765 2024-09-04T19:50:34.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:50:34.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:50:34.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-373662154765 2024-09-04T19:50:34.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:50:34.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154765 2024-09-04T19:50:34.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 373662154765' 2024-09-04T19:50:34.724 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 373662154765 2024-09-04T19:50:34.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:50:35.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154763 -lt 373662154765 2024-09-04T19:50:35.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:50:36.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:50:36.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:50:36.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154765 -lt 373662154765 2024-09-04T19:50:36.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:50:36.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056652 2024-09-04T19:50:36.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:50:36.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-04T19:50:36.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056652 2024-09-04T19:50:36.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:50:36.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056652 2024-09-04T19:50:36.533 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 386547056652 2024-09-04T19:50:36.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056652' 2024-09-04T19:50:36.533 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-04T19:50:36.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056652 -lt 386547056652 2024-09-04T19:50:36.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:50:36.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-399431958538 2024-09-04T19:50:36.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:50:36.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-04T19:50:36.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-399431958538 2024-09-04T19:50:36.936 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:50:36.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958538 2024-09-04T19:50:36.937 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 399431958538 2024-09-04T19:50:36.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 399431958538' 2024-09-04T19:50:36.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-04T19:50:37.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958539 -lt 399431958538 2024-09-04T19:50:37.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:50:37.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-416611827721 2024-09-04T19:50:37.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:50:37.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-04T19:50:37.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-416611827721 2024-09-04T19:50:37.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:50:37.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827721 2024-09-04T19:50:37.590 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 416611827721 2024-09-04T19:50:37.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 416611827721' 2024-09-04T19:50:37.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-04T19:50:37.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827722 -lt 416611827721 2024-09-04T19:50:37.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:50:37.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:50:37.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:50:38.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-04T19:50:38.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:50:38.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:50:38.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-04T19:50:38.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:50:38.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:50:38.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:50:39.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:50:39.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:50:39.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:50:39.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:50:39.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:50:39.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:50:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:50:39.623 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:50:39.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:50:39.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:50:39.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:50:39.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:50:39.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:50:39.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:50:39.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:50:39.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:50:39.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:50:39.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:50:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:50:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:50:40.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:50:40.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:50:40.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:50:40.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:50:40.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:50:40.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:50:40.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:50:40.111 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:50:40.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:50:40.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:50:40.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:50:40.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:50:40.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:50:40.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:50:40.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:50:40.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:50:40.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:50:40.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:50:40.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:50:40.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.117 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:50:40.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:50:40.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:50:40.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-04T19:50:40.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:50:40.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96820 2024-09-04T19:50:40.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:50:40.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-04T19:50:40.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:50:40.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-04T19:50:40.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-04T19:50:40.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-04T19:50:40.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-04T19:50:40.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-04T19:50:40.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-04T19:50:40.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-04T19:50:40.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-04T19:50:40.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-04T19:50:40.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:50:40.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:50:40.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:50:40.527 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:50:40.527 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.527 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:50:40.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:50:40.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:50:40.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:50:40.754 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:50:40.755 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:50:40.755 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:50:40.755 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.756 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.756 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:50:40.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:50:40.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get fsid 2024-09-04T19:50:40.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:50:40.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:50:40.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:50:40.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:50:40.867 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:50:40.867 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:50:40.867 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:50:40.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:50:40.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get mon_host 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-04T19:50:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:50:41.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:50:41.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:50:41.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:50:41.902 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:50:41.902 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:41.902 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:41.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:50:41.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:50:41.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:50:41.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:50:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:50:41.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:50:41.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be30e600-1ef0-44e0-9b57-e019e2bb2111 2024-09-04T19:50:41.984 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 be30e600-1ef0-44e0-9b57-e019e2bb2111 2024-09-04T19:50:41.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 be30e600-1ef0-44e0-9b57-e019e2bb2111' 2024-09-04T19:50:41.984 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:50:42.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASuthmOrdTABAA5A9ykDwfGy81nZ1Ps57FbA== 2024-09-04T19:50:42.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASuthmOrdTABAA5A9ykDwfGy81nZ1Ps57FbA=="}' 2024-09-04T19:50:42.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be30e600-1ef0-44e0-9b57-e019e2bb2111 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:50:42.308 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:50:42.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:50:42.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASuthmOrdTABAA5A9ykDwfGy81nZ1Ps57FbA== --osd-uuid be30e600-1ef0-44e0-9b57-e019e2bb2111 2024-09-04T19:50:42.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:42.367+0000 7f25100887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:42.376 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:42.371+0000 7f25100887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:42.377 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:42.371+0000 7f25100887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:42.378 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:42.371+0000 7f25100887c0 -1 bdev(0x55d3b7a80a80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:50:42.378 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:42.371+0000 7f25100887c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-04T19:50:50.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-04T19:50:50.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:50:50.615 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:50:50.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:50:50.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:50:51.176 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:50:51.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:50:51.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:50:51.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:50:51.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:50:51.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:50:51.213 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:51.211+0000 7f22695b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:51.215 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:51.215+0000 7f22695b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:51.216 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:51.215+0000 7f22695b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:50:51.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:51.598 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:50:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:50:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:50:52.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:52.709 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:52.707+0000 7f22695b87c0 -1 Falling back to public interface 2024-09-04T19:50:53.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:53.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:53.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:50:53.014 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:50:53.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:53.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:50:53.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:54.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:54.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:54.419 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:50:54.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:50:54.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:50:54.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:54.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:55.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:55.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:55.836 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:50:55.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:50:55.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:55.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:50:56.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:57.019 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:50:57.019+0000 7f22695b87c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:50:57.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:57.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:57.255 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:50:57.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:50:57.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:57.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:50:58.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:50:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:50:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:50:59.777 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:50:59.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:50:59.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:50:59.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:00.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:01.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:01.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:01.515 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:51:01.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:51:01.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:01.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:01.967 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/442120240,v1:127.0.0.1:6803/442120240] [v2:127.0.0.1:6804/442120240,v1:127.0.0.1:6805/442120240] exists,up be30e600-1ef0-44e0-9b57-e019e2bb2111 2024-09-04T19:51:01.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:51:01.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:51:01.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:51:01.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:51:01.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=145709 2024-09-04T19:51:01.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-04T19:51:06.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-04T19:51:06.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-04T19:51:07.578 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'foo' already exists 2024-09-04T19:51:07.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:51:08.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-04T19:51:08.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-04T19:51:08.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-04T19:51:08.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-04T19:51:08.665 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:51:08.667 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-04T19:51:08.667 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_147000 2024-09-04T19:51:08.703 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:51:08.703 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:51:09.703 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 17 1 3.99957 4 0.308909 0.308909 2024-09-04T19:51:10.703 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 33 17 33.9954 64 1.09426 1.23284 2024-09-04T19:51:11.703 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 49 33 43.9938 64 0.87659 1.06982 2024-09-04T19:51:12.703 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 65 49 48.9925 64 1.07729 1.06604 2024-09-04T19:51:13.703 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 81 65 51.9921 64 1.19505 1.06097 2024-09-04T19:51:14.704 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 97 81 53.9915 64 1.05204 1.07755 2024-09-04T19:51:15.704 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 113 97 55.4198 64 1.29504 1.12438 2024-09-04T19:51:16.704 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 129 113 56.491 64 0.901835 1.07804 2024-09-04T19:51:17.704 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 145 129 57.3243 64 0.994024 1.07562 2024-09-04T19:51:18.704 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 161 145 57.9908 64 0.651091 1.03405 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 10.3311 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 161 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4194304 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4194304 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 62.336 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 18.9737 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 64 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 4 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 15 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 4.74342 2024-09-04T19:51:19.705 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 16 2024-09-04T19:51:19.706 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 1 2024-09-04T19:51:19.706 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 1.01892 2024-09-04T19:51:19.706 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.208679 2024-09-04T19:51:19.706 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 1.38806 2024-09-04T19:51:19.706 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.308909 2024-09-04T19:51:19.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-04T19:51:29.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-04T19:51:29.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-04T19:51:29.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1059061760 2024-09-04T19:51:29.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-04T19:51:29.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=73924608 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 73924608 -eq 0 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 73924608 -gt 0 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 145709 2024-09-04T19:51:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-04T19:51:31.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 145709 2024-09-04T19:51:31.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (145709) - No such process 2024-09-04T19:51:31.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-04T19:51:31.551 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:51:31.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-04T19:51:31.593 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:51:41.723 INFO:tasks.workunit.client.0.smithi174.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-04T19:51:41.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-04T19:51:41.781 INFO:tasks.workunit.client.0.smithi174.stdout:inferring bluefs devices from bluestore path 2024-09-04T19:51:43.852 INFO:tasks.workunit.client.0.smithi174.stdout:1 : device size 0xffffe000 : using 0x43a98000(1.1 GiB) 2024-09-04T19:51:43.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-04T19:51:46.495 INFO:tasks.workunit.client.0.smithi174.stdout:fsck success 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-04T19:51:46.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:51:46.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:51:46.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:51:46.542 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:51:46.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-04T19:51:46.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:51:46.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-04T19:51:46.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-04T19:51:46.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-04T19:51:46.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-04T19:51:46.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-04T19:51:46.579 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:51:46.578+0000 7f12d202b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:51:46.582 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:51:46.578+0000 7f12d202b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:51:46.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:51:46.582+0000 7f12d202b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:51:46.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-04T19:51:46.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:51:46.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:51:46.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:46.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:47.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:47.641 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:51:47.638+0000 7f12d202b7c0 -1 Falling back to public interface 2024-09-04T19:51:48.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:48.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:48.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:51:48.415 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:51:48.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:48.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:48.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:49.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:49.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:49.866 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:51:49.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:51:49.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:49.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:50.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:51.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:51.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:51.304 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:51:51.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:51:51.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:51.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:51.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:52.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:52.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:52.733 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:51:52.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:51:52.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:52.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:53.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:54.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:54.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:54.226 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:51:54.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:51:54.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:54.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:54.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:55.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:55.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:55.636 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:51:55.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:51:55.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:55.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:55.670 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:51:55.666+0000 7f12d202b7c0 -1 osd.0 10 log_to_monitors true 2024-09-04T19:51:56.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:57.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:57.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:57.389 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-04T19:51:57.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-04T19:51:57.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:57.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:57.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:51:58.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:51:58.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:51:58.804 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-04T19:51:58.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-04T19:51:58.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:51:58.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:51:59.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:00.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:00.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:00.407 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-04T19:52:00.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-04T19:52:00.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:00.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:00.855 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/1136701391,v1:127.0.0.1:6803/1136701391] [v2:127.0.0.1:6804/1136701391,v1:127.0.0.1:6805/1136701391] exists,up be30e600-1ef0-44e0-9b57-e019e2bb2111 2024-09-04T19:52:00.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:52:00.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:52:00.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:52:00.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:52:00.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=148231 2024-09-04T19:52:00.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-04T19:52:00.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-04T19:52:00.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-04T19:52:00.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-04T19:52:00.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-04T19:52:00.858 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-04T19:52:00.858 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-04T19:52:00.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-04T19:52:00.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-04T19:52:00.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-04T19:52:00.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-04T19:52:01.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-04T19:52:01.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-04T19:52:01.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-04T19:52:01.364 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-04T19:52:01.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574852 2024-09-04T19:52:01.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574852 2024-09-04T19:52:01.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-55834574852' 2024-09-04T19:52:01.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-04T19:52:01.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-55834574852 2024-09-04T19:52:01.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-04T19:52:01.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-04T19:52:01.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-55834574852 2024-09-04T19:52:01.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-04T19:52:01.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574852 2024-09-04T19:52:01.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-04T19:52:01.543 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 55834574852 2024-09-04T19:52:01.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:52:01.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 55834574852 2024-09-04T19:52:01.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-04T19:52:02.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-04T19:52:02.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-04T19:52:03.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-04T19:52:03.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-04T19:52:03.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:52:03.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-04T19:52:03.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-04T19:52:03.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-04T19:52:03.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-04T19:52:04.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-04T19:52:04.383 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-04T19:52:04.383 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-04T19:52:04.383 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-04T19:52:04.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:52:04.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:52:05.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:52:05.040 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:52:05.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:52:05.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:52:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:52:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:52:05.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:52:05.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:52:05.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:52:05.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:52:05.045 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:52:05.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:52:05.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:52:05.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:52:05.159 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:52:05.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-04T19:52:05.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-04T19:52:05.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:52:05.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:52:05.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:52:05.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:52:05.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:52:05.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:52:05.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:52:05.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:52:05.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:52:05.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:52:05.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:52:05.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:52:05.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:52:05.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:52:05.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:52:05.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:52:05.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:52:05.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:52:05.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-04T19:52:05.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:52:05.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:52:05.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:52:05.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:52:05.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:52:05.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-04T19:52:05.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-04T19:52:05.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.96820 2024-09-04T19:52:05.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-04T19:52:05.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-04T19:52:05.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-04T19:52:05.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=352858d4-95e6-4514-b11b-6ec40e28d047 --auth-supported=none ' 2024-09-04T19:52:05.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-04T19:52:05.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-04T19:52:05.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-04T19:52:05.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.508 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.509 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:52:05.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-04T19:52:05.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-04T19:52:05.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-04T19:52:05.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:52:05.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:52:05.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-04T19:52:05.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:52:05.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:52:05.712 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get fsid 2024-09-04T19:52:05.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-04T19:52:05.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-04T19:52:05.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-04T19:52:05.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-04T19:52:05.829 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-04T19:52:05.829 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-04T19:52:05.829 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-04T19:52:05.829 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.96820/ceph-mon.a.asok 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-04T19:52:05.830 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.96820/ceph-mon.a.asok config get mon_host 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-04T19:52:05.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-04T19:52:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-04T19:52:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:52:06.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:52:06.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:52:06.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:06.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:06.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:52:06.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:52:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-04T19:52:06.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=352858d4-95e6-4514-b11b-6ec40e28d047 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-04T19:52:06.896 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-04T19:52:06.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-04T19:52:06.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-04T19:52:06.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-04T19:52:06.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-04T19:52:06.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3de1d629-7c62-4680-9693-c07d9d8831c0 2024-09-04T19:52:06.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3de1d629-7c62-4680-9693-c07d9d8831c0' 2024-09-04T19:52:06.899 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 3de1d629-7c62-4680-9693-c07d9d8831c0 2024-09-04T19:52:06.899 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-04T19:52:06.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmuthm34reNhAAjWJE6s5TpHopikrOC3uugQ== 2024-09-04T19:52:06.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmuthm34reNhAAjWJE6s5TpHopikrOC3uugQ=="}' 2024-09-04T19:52:06.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3de1d629-7c62-4680-9693-c07d9d8831c0 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:52:07.203 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:52:07.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-04T19:52:07.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=352858d4-95e6-4514-b11b-6ec40e28d047 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmuthm34reNhAAjWJE6s5TpHopikrOC3uugQ== --osd-uuid 3de1d629-7c62-4680-9693-c07d9d8831c0 2024-09-04T19:52:07.263 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:07.262+0000 7fea6f9777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:07.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:07.270+0000 7fea6f9777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:07.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:07.270+0000 7fea6f9777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:07.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:07.270+0000 7fea6f9777c0 -1 bdev(0x564d67eb4a80 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-04T19:52:07.273 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:07.270+0000 7fea6f9777c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-04T19:52:15.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-04T19:52:15.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-04T19:52:15.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-04T19:52:15.038 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-04T19:52:15.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-04T19:52:15.619 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-04T19:52:15.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-04T19:52:15.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8a6520a2-3999-499c-bc45-95453d0ea966 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=5e2102ab-0f4a-4b25-917f-fc4d20727f41 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=352858d4-95e6-4514-b11b-6ec40e28d047 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.96820/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-04T19:52:15.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-04T19:52:15.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-04T19:52:15.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-04T19:52:15.662 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:15.658+0000 7f04a846d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:15.665 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:15.662+0000 7f04a846d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:15.667 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:15.666+0000 7f04a846d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:16.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-04T19:52:16.081 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-04T19:52:16.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:16.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:16.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:17.417 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:17.414+0000 7f04a846d7c0 -1 Falling back to public interface 2024-09-04T19:52:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:17.501 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-04T19:52:17.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-04T19:52:17.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:17.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:17.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:18.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:18.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:18.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-04T19:52:18.918 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-04T19:52:18.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:18.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:19.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:20.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:20.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:20.399 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-04T19:52:20.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-04T19:52:20.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:20.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:20.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:21.394 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:52:21.390+0000 7f04a846d7c0 -1 osd.0 0 log_to_monitors true 2024-09-04T19:52:21.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:21.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:21.828 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-04T19:52:21.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-04T19:52:21.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:21.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:22.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:23.766 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-04T19:52:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-04T19:52:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:25.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-04T19:52:26.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-04T19:52:26.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-04T19:52:26.223 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-04T19:52:26.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-04T19:52:26.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-04T19:52:26.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-04T19:52:26.662 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1702658213,v1:127.0.0.1:6803/1702658213] [v2:127.0.0.1:6804/1702658213,v1:127.0.0.1:6805/1702658213] exists,up 3de1d629-7c62-4680-9693-c07d9d8831c0 2024-09-04T19:52:26.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-04T19:52:26.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-04T19:52:26.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-04T19:52:26.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-04T19:52:26.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=151012 2024-09-04T19:52:26.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-04T19:52:31.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-04T19:52:31.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-04T19:52:32.284 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'foo' already exists 2024-09-04T19:52:32.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-04T19:52:33.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-04T19:52:33.358 INFO:tasks.workunit.client.0.smithi174.stdout:hints = 1 2024-09-04T19:52:33.358 INFO:tasks.workunit.client.0.smithi174.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-04T19:52:33.358 INFO:tasks.workunit.client.0.smithi174.stdout:Object prefix: benchmark_data_smithi174_152047 2024-09-04T19:52:33.358 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:52:33.358 INFO:tasks.workunit.client.0.smithi174.stdout: 0 0 0 0 0 0 - 0 2024-09-04T19:52:34.360 INFO:tasks.workunit.client.0.smithi174.stdout: 1 16 128 112 0.43745 0.4375 0.117881 0.14004 2024-09-04T19:52:35.358 INFO:tasks.workunit.client.0.smithi174.stdout: 2 16 275 259 0.505803 0.574219 0.0922175 0.120781 2024-09-04T19:52:36.358 INFO:tasks.workunit.client.0.smithi174.stdout: 3 16 403 387 0.503849 0.5 0.100443 0.119964 2024-09-04T19:52:37.359 INFO:tasks.workunit.client.0.smithi174.stdout: 4 16 560 544 0.531179 0.613281 0.108925 0.116269 2024-09-04T19:52:38.359 INFO:tasks.workunit.client.0.smithi174.stdout: 5 16 675 659 0.514774 0.449219 0.117544 0.118878 2024-09-04T19:52:39.359 INFO:tasks.workunit.client.0.smithi174.stdout: 6 16 832 816 0.53118 0.613281 0.118032 0.11708 2024-09-04T19:52:40.359 INFO:tasks.workunit.client.0.smithi174.stdout: 7 16 960 944 0.526718 0.5 0.100973 0.118408 2024-09-04T19:52:41.359 INFO:tasks.workunit.client.0.smithi174.stdout: 8 16 1056 1040 0.507745 0.375 0.117965 0.122628 2024-09-04T19:52:42.359 INFO:tasks.workunit.client.0.smithi174.stdout: 9 16 1171 1155 0.501235 0.449219 0.133883 0.123757 2024-09-04T19:52:43.359 INFO:tasks.workunit.client.0.smithi174.stdout: 10 16 1312 1296 0.506182 0.550781 0.116384 0.122693 2024-09-04T19:52:44.360 INFO:tasks.workunit.client.0.smithi174.stdout: 11 16 1440 1424 0.505613 0.5 0.0927175 0.123332 2024-09-04T19:52:45.360 INFO:tasks.workunit.client.0.smithi174.stdout: 12 16 1587 1571 0.511323 0.574219 0.125448 0.121649 2024-09-04T19:52:46.360 INFO:tasks.workunit.client.0.smithi174.stdout: 13 16 1712 1696 0.509543 0.488281 0.100393 0.122289 2024-09-04T19:52:47.360 INFO:tasks.workunit.client.0.smithi174.stdout: 14 16 1859 1843 0.514156 0.574219 0.108999 0.121019 2024-09-04T19:52:48.360 INFO:tasks.workunit.client.0.smithi174.stdout: 15 16 2000 1984 0.516592 0.550781 0.125723 0.120807 2024-09-04T19:52:49.360 INFO:tasks.workunit.client.0.smithi174.stdout: 16 16 2144 2128 0.519458 0.5625 0.109278 0.119959 2024-09-04T19:52:50.361 INFO:tasks.workunit.client.0.smithi174.stdout: 17 16 2272 2256 0.518309 0.5 0.134488 0.120257 2024-09-04T19:52:51.361 INFO:tasks.workunit.client.0.smithi174.stdout: 18 16 2400 2384 0.517287 0.5 0.117484 0.120428 2024-09-04T19:52:52.361 INFO:tasks.workunit.client.0.smithi174.stdout: 19 16 2531 2515 0.51699 0.511719 0.117508 0.120489 2024-09-04T19:52:53.361 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-04T19:52:53.361896+0000 min lat: 0.0670963 max lat: 0.442396 avg lat: 0.119335 2024-09-04T19:52:53.361 INFO:tasks.workunit.client.0.smithi174.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-04T19:52:53.361 INFO:tasks.workunit.client.0.smithi174.stdout: 20 16 2691 2675 0.522386 0.625 0.100721 0.119335 2024-09-04T19:52:54.361 INFO:tasks.workunit.client.0.smithi174.stdout: 21 16 2816 2800 0.520758 0.488281 0.118385 0.119873 2024-09-04T19:52:55.361 INFO:tasks.workunit.client.0.smithi174.stdout: 22 16 2960 2944 0.522652 0.5625 0.0926412 0.11924 2024-09-04T19:52:56.361 INFO:tasks.workunit.client.0.smithi174.stdout: 23 16 3088 3072 0.521664 0.5 0.100789 0.119559 2024-09-04T19:52:57.361 INFO:tasks.workunit.client.0.smithi174.stdout: 24 16 3248 3232 0.525966 0.625 0.10873 0.118742 2024-09-04T19:52:58.362 INFO:tasks.workunit.client.0.smithi174.stdout: 25 16 3347 3331 0.520394 0.386719 0.167544 0.119612 2024-09-04T19:52:59.362 INFO:tasks.workunit.client.0.smithi174.stdout: 26 16 3491 3475 0.52201 0.5625 0.100798 0.119422 2024-09-04T19:53:00.362 INFO:tasks.workunit.client.0.smithi174.stdout: 27 16 3632 3616 0.523072 0.550781 0.092387 0.119291 2024-09-04T19:53:01.362 INFO:tasks.workunit.client.0.smithi174.stdout: 28 16 3779 3763 0.524896 0.574219 0.0922452 0.118864 2024-09-04T19:53:02.362 INFO:tasks.workunit.client.0.smithi174.stdout: 29 16 3904 3888 0.52363 0.488281 0.125324 0.119196 2024-09-04T19:53:03.362 INFO:tasks.workunit.client.0.smithi174.stdout: 30 16 4035 4019 0.523231 0.511719 0.108998 0.119231 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Total time run: 30.1 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Total writes made: 4035 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Write size: 4096 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Object size: 4096 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Bandwidth (MB/sec): 0.523646 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Bandwidth: 0.0644111 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Max bandwidth (MB/sec): 0.625 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Min bandwidth (MB/sec): 0.375 2024-09-04T19:53:04.363 INFO:tasks.workunit.client.0.smithi174.stdout:Average IOPS: 134 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev IOPS: 16.4893 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Max IOPS: 160 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Min IOPS: 96 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Average Latency(s): 0.11919 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Stddev Latency(s): 0.0290661 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Max latency(s): 0.442396 2024-09-04T19:53:04.364 INFO:tasks.workunit.client.0.smithi174.stdout:Min latency(s): 0.0661665 2024-09-04T19:53:04.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-04T19:53:09.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-04T19:53:09.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-04T19:53:09.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-04T19:53:09.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-04T19:53:09.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-04T19:53:09.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-04T19:53:09.615 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-04T19:53:09.615 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-04T19:53:09.615 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-04T19:53:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4278333440 2024-09-04T19:53:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151012 2024-09-04T19:53:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-04T19:53:10.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151012 2024-09-04T19:53:10.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (151012) - No such process 2024-09-04T19:53:10.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-04T19:53:11.225 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already down. 2024-09-04T19:53:11.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-04T19:53:11.275 INFO:tasks.workunit.client.0.smithi174.stdout:allocmap bluestore.allocmap 2024-09-04T19:53:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:53:14.049+0000 7fe100ba9ac0 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-04T19:53:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:53:14.049+0000 7fe100ba9ac0 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-04T19:53:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-04T19:53:14.049+0000 7fe100ba9ac0 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-04T19:53:14.627 INFO:tasks.workunit.client.0.smithi174.stderr:allocmap failed: (1) Operation not permitted 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-bluefs-volume-ops 1 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-09-04T19:53:14.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-04T19:53:14.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-04T19:53:14.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-04T19:53:14.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-04T19:53:14.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-04T19:53:14.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-04T19:53:14.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-04T19:53:14.747 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-04T19:53:14.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-04T19:53:14.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-04T19:53:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-04T19:53:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-04T19:53:14.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-04T19:53:14.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:53:14.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-04T19:53:14.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-04T19:53:14.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-04T19:53:14.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:53:14.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-04T19:53:14.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-09-04T19:53:14.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-04T19:53:14.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-04T19:53:14.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-04T19:53:14.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-04T19:53:14.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-04T19:53:14.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.96820 2024-09-04T19:53:14.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.96820 2024-09-04T19:53:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T19:53:14.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-04T19:53:14.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-04T19:53:14.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-09-04T19:53:14.764 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-04T19:53:14.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-04T19:53:15.112 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-04T19:53:15.294 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c3c73d91e9914b9f93f6aeaeac48bb8d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ljflores_ceph_055b029e365e266078e29644ed3ff217dfe73d04/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-04T19:53:15.297 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-04T19:53:15.306 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-04T19:53:15.306 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-04T19:53:15.346 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-04T19:53:15.355 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-04T19:53:15.356 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-04T19:53:15.377 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-04T19:53:15.377 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-04T19:53:15.377 INFO:teuthology.orchestra.run.smithi174.stdout:*hv01.front.sepi 67.205.162.81 3 u 13 64 377 0.093 -1.184 0.120 2024-09-04T19:53:15.378 INFO:teuthology.orchestra.run.smithi174.stdout:+hv02.front.sepi 63.231.80.2 3 u 15 64 377 0.100 -1.456 0.216 2024-09-04T19:53:15.378 INFO:teuthology.orchestra.run.smithi174.stdout:+hv03.front.sepi 74.6.168.72 3 u 21 64 377 0.083 -0.974 0.102 2024-09-04T19:53:15.378 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-04T19:53:15.379 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-04T19:53:15.387 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-04T19:53:15.387 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-04T19:53:15.394 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-04T19:53:15.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-04T19:53:15.409 INFO:teuthology.task.internal:Duration was 2897.281523 seconds 2024-09-04T19:53:15.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-04T19:53:15.417 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-04T19:53:15.417 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-04T19:53:15.464 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-04T19:53:15.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-04T19:53:15.464 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-04T19:53:15.516 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-04T19:53:15.517 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-04T19:53:15.602 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-04T19:53:15.603 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-04T19:53:15.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-04T19:53:15.718 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-04T19:53:15.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-04T19:53:15.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-04T19:53:15.765 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:53:15.785 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-04T19:53:15.801 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-04T19:53:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T19:53:15.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-04T19:53:15.856 INFO:teuthology.task.internal:Transferring archived files... 2024-09-04T19:53:15.858 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-09-04_18:54:56-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7889511/remote/smithi174 2024-09-04T19:53:15.859 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-04T19:53:18.824 INFO:teuthology.task.internal:Removing archive directory... 2024-09-04T19:53:18.825 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-04T19:53:18.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-04T19:53:18.892 INFO:teuthology.task.internal:Not uploading archives. 2024-09-04T19:53:18.892 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-04T19:53:18.906 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-04T19:53:18.906 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-04T19:53:18.932 INFO:teuthology.orchestra.run.smithi174.stdout: 658626 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 4 19:53 /home/ubuntu/cephtest 2024-09-04T19:53:18.932 INFO:teuthology.orchestra.run.smithi174.stdout: 788830 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 4 19:14 /home/ubuntu/cephtest/mnt.0 2024-09-04T19:53:18.932 INFO:teuthology.orchestra.run.smithi174.stdout: 790622 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 4 19:40 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-04T19:53:18.932 INFO:teuthology.orchestra.run.smithi174.stdout: 793456 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 4 19:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-04T19:53:18.933 INFO:teuthology.orchestra.run.smithi174.stdout: 805872 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 4 19:53 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-04T19:53:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-04T19:53:18.934 INFO:teuthology.orchestra.run.smithi174.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-04T19:53:18.934 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-04T19:53:18.939 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-04T19:53:18.948 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-04T19:53:18.958 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi174 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-04T19:53:18.962 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 2897.281522989273 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi174 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055b029e365e266078e29644ed3ff217dfe73d04 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_lflores@folio02 sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c3c73d91e9914b9f93f6aeaeac48bb8d status: fail success: false 2024-09-04T19:53:18.962 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-04T19:53:19.036 INFO:teuthology.run:FAIL